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Executive Summary 



In a 1999 National Research Council report, the committee wrote: 

One striking fact is that the complex world of education — unlike defense, health care, or industrial 
production — does not rest on a strong research base. In no other field are personal experience and 
ideology so frequently relied on to make policy choices, and in no other field is the research base so 
inadequate and little used.* 

Others, including members of Congress, shared the view that education research had not provided education 
policymakers and practitioners with the information and tools they needed to improve education in our country. 
When the Institute of Education Sciences (Institute) was established in November 2002, many in the education 
research and policy community cried deja vu. As some observed, why would anyone expect the Institute to 
accomplish what its predecessors — the National Institute of Education and the Office of Educational Research and 
Improvement — had not? On the basis of its evaluation, the members of the National Board for Education Sciences 
(Board) conclude that in a relatively brief period of time, the Institute has made exceptional progress in improving the 
rigor and relevance of education research in our nation. Under the leadership of its first director, Grover J. Whitehurst, 
the Institute has accomplished what many believed could not be done. 

The framework for the Institute’s nonideological, high-quality work was wisely established by Congress in the 
Education Sciences Reform Act of 2002 (ESRA). Because that Act has generated such strong results, the Board is 
recommending its rapid reauthorization, with a set of modest amendments meant to improve its clarity and make 
it even stronger. The Board recognizes that transformation of education into an evidence-based field is an enormous 
task. It will need to involve everyone from federal and state policymakers to local education leaders, administrators, 
teachers, and parents. Over the past 6 years, a new direction has been set for education research. We now need to stay 
on course to arrive at this destination. 

This report presents the Board’s evaluation of the Institute. The Board examined the ways in which and the extent to 
which the Institute has been successful in advancing the rigor and improving the relevance of education research, and 
facilitating evidence-based decisionmaking. 

Advancing the Rigor of Education Research 

The Institute was established to provide rigorous evidence on which to ground education practice and policy. In its 
first 6 years, the Institute has made exceptional progress in improving the rigor of education research and evaluation. 
By design, the goal of improved rigor has driven the majority of staff activity to date, under the assumption that the 
threshold condition for making education an evidence-based field is producing findings that can be trusted. The 
Board concurs with the strategic decision to emphasize rigor. Our judgment of exceptional progress is based on the 
following evidence: (1) the high standards reflected in the peer review system; (2) the strong external ratings of the 
quality of the funded research grants; and (3) the high quality of the research designs of the evaluations contracted 



* National Research Council. (1999). Improving Student Learning: A Strategic Plan for Education Research and Its Utilization. Committee on a 
Feasibility Study for a Strategic Education Research Program, Commission on Behavioral and Social Sciences in Education. Washington, DC: 
Author. 
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through the Institute’s National Center for Education Evaluation and Regional Assistance (NCEE). To build the 
nation’s capacity to conduct rigorous education research, the Institute has launched predoctoral and postdoctoral 
research training programs. Based on the GRE scores of the predoctoral fellows and the research productivity of the 
predoctoral and postdoctoral fellows, as evidenced by the numbers of research publications to date, the Institute has 
made substantial progress in developing a new generation of education scientists who are well-equipped to conduct 
high-quality research. Finally, in light of concerns that the Institute has narrowly focused its research funding on 
projects that use experimental methods to evaluate the effectiveness of interventions, the Board examined the 
Institute’s funding announcements and diversity of the research grant portfolio with respect to research questions and 
research methods employed. The Board found that the Institute clearly requests research projects that are diverse in 
purpose (e.g., exploring malleable factors, developing and validating assessments) and in methodological requirements 
(e.g., correlational, descriptive, observational, quasi-experimental, and experimental methodologies) and has developed 
a diverse research portfolio in which roughly one-fourth of the projects are experimental or quasi-experimental 
evaluations of the impact of interventions on education outcomes. 

Improving the Relevance of Education Research 

No matter how technically sound research activities might be, if they do not address the issues and questions that 
are of concern to education policymakers and practitioners, the research will not be used to inform education policy 
and practice. The Institute has made substantial progress in improving the relevance and usefulness of education 
research, evaluation, and statistics. This is an area that has drawn increased attention from staff as the rigor of the 
work improved, and the Board agrees with this strategic shift. It should be continued in the future. Our judgment 
of substantial progress is based on the following: (I) satisfactory ratings of the relevance of funded research projects 
by education leaders and administrators; (2) establishment of long-term focused programs of research that address 
fundamental education issues in our nation (e.g., improving reading, writing, mathematics, and science achievement); 
(3) the work of the National Research and Development Centers in key policy areas; (4) dramatic improvements in 
the timeliness of the release of data from the Institute’s National Center for Education Statistics (NCES); (5) high 
ratings of the relevance of NCES reports; and (6) increased efforts to improve the timeliness of the release of NCEE 
evaluation reports. 

Facilitating Evidence-Based Decisionmaking 

Generating rigorous and relevant research is a necessary step to the transformation of education into an evidence-based 
field, but the Institute cannot stop there. The Institute must disseminate the knowledge it produces in ways that enable 
education policymakers and practitioners to use that information. In this arena, the Institute has also made substantial 
progress, particularly in the last 2 years, and this area should continue to receive sustained staff attention in the future. 
Our finding of substantial progress is based upon the quality and use of the systems and programs the Institute has 
created (e.g.. What Works Clearinghouse [WWC], College Navigator) or revamped (e.g.. Education Resources 
Information Center [ERIC]) to disseminate practical information to education leaders, practitioners, parents, and 
students. The WWC practice guide Organizing Instruction and Study to Improve Student Learning has been downloaded 
nearly 30,000 times since its release last September; the Turning Around Chronically Low-Performing Schools practice 
guide has been downloaded nearly 20,000 times since its release in May 2008. The College Navigator was rated by 
Money Magazine as one of the best places to start a college search. Our finding was also supported by examination of 
the programs the Institute has established to help education leaders and decisionmakers obtain better data and become 
better consumers of education research, such as the Statewide Eongitudinal Data Systems, the National Forum on 
Education Statistics, and the NCES Summer Data Conference. 
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Despite the Board’s view that the Institute has been instrumental in improving the rigor, relevance, and accessibility 
of federally funded educational research, much remains to be done to institutionalize the gains made and build on 
them. For this reason, the full report includes recommendations from the Board regarding the details of the Institute’s 
work and the reauthorization of the ESRA. However, the main message from our evaluation is that going forward, 
the Institute should maintain the direction that Congress articulated and that the Institute’s leadership and staff have 
executed so well. 
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Introduction 



The Institute of Education Sciences (Institute) within the U.S. Department of Education (ED) was authorized by the 
Education Sciences Reform Act of 2002 (ESRA). ESRA established the National Board for Education Sciences (Board) 
as a board of directors for the Institute. Among other duties, the Board is responsible for providing to Congress, the 
Secretary of Education, and the Director of the Institute a final report that includes recommendations regarding any 
actions that may be taken to enhance the ability of the Institute to carry out its priorities and missions. To meet this 
obligation, the Board has undertaken an evaluation of the Institute and formulated recommendations to improve the 
Institute’s ability to execute its responsibilities and fulfill its mission. 

The Board evaluation addressed three basic questions: 

1 . To what extent, and in which ways, has the Institute been successful in advancing the rigor of education 
research? 

2. To what extent, and in which ways, has the Institute increased the relevance and usefulness of education 
research? 

3. To what extent, and in which ways, has the Institute facilitated evidence-based decisionmaking? 

This report provides a summary of the Board’s evaluation of the Institute, along with recommendations in the 
form of a mark-up of ESRA and 10 resolutions that the Board has passed since its first meeting in February 2005. 

We begin with a brief overview of the Institute’s mission and priorities, provide a summary and evaluation of the 
accomplishments of the Office of the Director and the four National Education Centers, and conclude with the 
Board’s recommendations. 



National Board for Education Sciences 5-Year 



Report, 2003 Through 2008 



1 



I. Institute of Education Sciences: 

Mission and Priorities 

In a 1999 National Research Council report, the committee wrote: 

One striking fact is that the complex world of education — unlike defense, health care, or industrial 
production — does not rest on a strong research base. In no other field are personal experience and 
ideology so frequently relied on to make policy choices, and in no other field is the research base so 
inadequate and little used.' 

Others, including members of Congress, shared the view that education research had not provided education 
policymakers and practitioners with the information and tools they needed to improve education in our country. It 
was in this context that the Institute was established to transform education research into an enterprise that provides 
rigorous evidence on which to ground education practice and policy. The Institute’s mission is to expand knowledge 
and provide information on (a) the condition of education; (b) practices that improve education outcomes; (c) the 
effectiveness of federal and other education programs; and (d) the educational needs of children with disabilities (as 
authorized in December 2004 in the Individuals with Disabilities Education Improvement Act). These responsibilities 
are carried out through the Institute’s four Centers: (1) National Center for Education Statistics (NCES); (2) National 
Center for Education Research (NCER); (3) National Center for Education Evaluation and Regional Assistance 
(NCEE); and (4) National Center for Special Education Research (NCSER). 

On September 6-7, 2005, the Board approved priorities for the Institute. The Institute’s overarching priority is 
research that contributes to improved academic achievement for all students, and particularly for those whose 
education prospects are hindered by inadequate education services and conditions associated with poverty, race/ 
ethnicity, limited English proficiency, disability, and family circumstance. With academic achievement as the major 
priority, the Institute focuses on outcomes that differ by periods of education. In the infancy and preschool period, the 
outcomes of interest are readiness for schooling and developmental outcomes for infants and toddlers with disabilities. 
In kindergarten through 12th grade, the priorities are the core academic outcomes of reading and writing (including 
reading and writing in the disciplines), mathematics, and science, as well as the behaviors and social skills that support 
learning in school and successful transitions to employment, independent living, and postsecondary education. At the 
postsecondary level, the priority is on enrollment in and completion of programs that prepare students for successful 
careers and lives. The same outcomes are emphasized for students with disabilities across each of these periods, and 
include the functional outcomes that improve educational and transitional results. The acquisition of basic skills by 
adults with low levels of education is also a priority. 

The long-term goals associated with the Institute’s priorities are fourfold: (a) to develop or identify a substantial 
number of programs, practices, policies, and approaches that enhance academic achievement and that can be 
widely deployed; (b) to identify what does not work, and what is problematic or inefficient, and thereby encourage 
innovation and further research; (c) to gain fundamental understanding of the processes that underlie variations in 
the effectiveness of education programs, practices, policies, and approaches; and (d) to develop delivery systems for 



’ National Research Council. (1999). Improving Student Learning: A Strategic Plan for Education Research and Its Utilization. Committee on a 
Feasibility Study for a Strategic Education Research Program, Commission on Behavioral and Social Sciences in Education. Washington, DC: 
Author. 
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the results of education research that will be routinely used by policymakers, educators, and the general public when 
making education decisions. 

A critical impetus for the creation of the Institute was the recognition by Congress that education research needed to 
become more rigorous and more directed to solving the problems identified by education leaders and practitioners. 
Now, nearly 6 years after the Institute was created, the Board reports on its examination of the extent to which the 
Institute has made progress in accomplishing its mission. 
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Evaluation of the Institute of Education 
Sciences 



A. Advancing the Rigor of Education Research 

A. 1 . Is the Institute raising the quality of research funded or conducted by the 
Department? 

To examine the extent to which and the ways in which the Institute has been successful in advancing the rigor of 
education research, the Board first considered the Institute’s peer review process, the quality of research funded by the 
Institute through its research grants, the evaluations contracted by NCEE, and the research conducted by the Regional 
Educational Eaboratories (REEs). Second, the Board considered the extent to which the Institute is building capacity 
in the field to conduct more rigorous research. 

In its first 6 years of operation, the Institute has made exceptional progress in improving the rigor of education 
research. The primary data leading to this assessment are the ratings of the quality of research grants funded, the high 
standards reflected in the peer review system and the funding rates, the high quality of the research designs for NCEE- 
funded evaluations, and comparisons of these indicators with practices prior to the formation of the Institute. In 
addition, the Board considered the quality and productivity of the predoctoral and postdoctoral fellows. 

A.l.a. Education and Special Education Research Projects 

To improve the scientific and technical quality of the research that the Institute funded, the Institute implemented a 
new scientific review system for education research grants. The legislation required the Director of the Institute “to 
establish necessary procedures for technical and scientific peer review of the activities of the Institute” to assure that 
“scientifically based research standards” are applied, among other things, to the funding of grant applications. The 
Institute established a scientific peer review system that is similar in many ways to the process of peer review at the 
National Institutes of Health (NIH). A key provision in this model is the separation between the program officers and 
administrators within the Institute who administer grant programs and work with applicants and grantees on the one 
hand, and those who are responsible for the peer review of applications for funding under these grant programs, on 
the other hand. The Standards and Review Office was created under the Deputy Director for Science and within the 
Office of the Director of the Institute. That office selects peer reviewers, determines review criteria, and manages the 
peer review panels that evaluate scientific merit of applications. In addition to creating a separate office for managing 
the scientific peer review process, the Institute has established a policy for selecting strong reviewers. Potential 
reviewers are identified primarily on the basis of the quality of the research they have conducted and published in 
scientific peer-reviewed journals and the degree to which they are experts in the research methods and subject matter 
that are relevant to the types of projects funded by the Institute.^ 

The quality of research projects funded through NCER under the guidelines of the Institute’s standards and 
procedures and the quality of evaluations supported by NCEE were compared to the quality of projects funded under 
the procedures and standards used by its predecessor, the Office of Educational Research and Improvement (OERI). 
Eminent senior scientists (e.g., distinguished professors in their institutions, editors of premier research journals, and 



^ Standards and Review Office, Peer Review of Grant Applications. Retrieved July 18, 2008, from 
http://ies.ed.gov/director/sro/peer review/application review.asp . 

National Board for Education Sciences 5-Year Report, 2003 Through 2008 ^ 0 # • 



5 



leading researchers in education) participated in this review, rating the quality of each project on a 9-point Likert- type 
scale in which 1 represents “very poor quality” and 9 represents “very high quality.” As can be seen in table 1 below, 
there was a marked improvement in the percentage of projects that were rated as being of high quality, from the 
previous OERI standards to the current Institute standards. 



Table 1 



Percentage of new research and evaluation projects that 
are deemed to be of high quality by an independent 
review panel of qualified scientists 



Year Percent 



2001 (OERI) 


36 


2002 (lES) 


50 


2003 (lES) 


70 


2004 (lES) 


70' 



’ In 2004, the scores of one review were extreme outliers — greater 
than 3.8 standard deviations below the average ratings of the other 
reviewers. If these scores were included, the percentage of new projects 
deemed to be of high quality would be 60 percent. 

NOTE: lES projects reported in table 1 include grants funded 
through NCER and evaluations contracted through NCEE. 
SOURCE: lES Standards and Review Office. 



After the scientific review procedures were deemed to be stable and capable of identifying high-quality proposals, 
the Institute eliminated the practice of sending newly funded proposals to a separate panel of scientists for a second 
(essentially quality-control) review. However, the Institute has tracked the overall ratings assigned by the scientific peer 
review panels to newly funded applications. Table 2 on the following page reports the average overall ratings of newly 
funded education research applications from 2003 to 2008;^ average ratings for all years are in the excellent range (1.5 
to 2.0). In 2006, NCSER was established as the fourth center within the Institute. Table 3 on the next page reports the 
average overall ratings of newly funded special education research applications from 2006 to 2008; average ratings for 
all years are in the excellent range. 



^ Overall ratings from 2002 are not included because the Institute changed the rating scale between 2002 and 2003. 
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Table 2 



Average overall ratings of newly funded education research projects and 
percentage of new education research 
projects that receive an average score of excellent or 
higher from the scientific peer review panel 

“Excellent” or 



“Outstanding” rating** 



Year 


Average rating* 


(Percent) 


2003 


1.71 


88 


2004 


1.63 


97 


2005 


1.65 


100 


2006 


1.78 


94 


2007 


1.83 


88 


2008 


1.83 


91 



*Rating scale: 1.0 to 1.4 = “Outstanding”; 1.5 to 2.0 = “Excellent”; 2.1 to 2.5 = “Very good” 
**Percentage of new research projects that receive an average score of excellent or higher 
from the scientific peer review panel. 

NOTE: Includes all grants from regular education research competitions, including research 
centers. 

SOURCE: lES Grants Administration Office documents. 



Table 3 



Average overall ratings of newly funded special education research 
projects and percentage of new special education research projects that 
receive an average score of excellent or higher from the scientific peer 

review panel 

“Excellent” or 
“Outstanding” rating** 



Year 


Average rating* 


(Percent) 


2006 


1.73 


93 


2007 


1.69 


100 


2008 


1.82 


85 



*Rating scale: 1.0 to 1.4 = “Outstanding”; 1.5 to 2.0 = “Excellent”; 2.1 to 2.5 = “Very good” 
**Percentage of new research projects that receive an average score of excellent or higher 
from the scientific peer review panel. 

NOTE; Includes all grants from regular special education research competitions, includes 
research centers. 

SOURCE: lES Grants Administration Office documents. 
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The Institute set a target of having 90 percent of the funded education research applications be rated as outstanding 
or excellent in technical merit. In setting a target of 90 percent, the Institute has elected to fund some proposals 
that scored slightly below excellent to fund proposals that address gaps in the research portfolio and for which the 
deficiencies in the proposals noted by the review panel were problems that could be remedied prior to conducting the 
research. For education research grants, with the exception of 2003 and 2007, the Institute has met this target. For 
special education research grants, the Institute met this target for 2 of 3 years. Given that the average overall ratings of 
newly funded education research and special education research grants have been in the excellent range for all years for 
both sets of grants, the Board is not concerned that the Institute was slightly under the 90 percent target three times. 
In the Board’s view, funding proposals with minor and easily fixed problems (as opposed to having researchers revise 
and resubmit proposals in the following year) enables the field to more quickly address important education problems. 

Beginning in 2002 with three research programs, the Institute has expanded its portfolio of research programs over 
the past 6 years. Notably, in 2006, the Institute held the first competitions for its special education research programs. 
Over this period, the Institute has substantially increased the number of responsive applications that it receives (i.e., 
the number of applications deemed meeting the requirements of the grant announcement). Figure 1 shows the 
increase in the total numbers of responsive applications across the education and special education research programs, 
as well as the numbers of grants awarded in the programs. 



Figure 1 



Total numbers of responsive research applications and grants for education and 
special education research competitions combined 




I Applications I Awards 



NOTE: Numbers reflect NCER and NCSER applications to and grants through regular education 
and special education research programs and Research and Development center competitions, but 
exclude applications to research training programs and unsolicited applications. 

SOURCE: lES Grants Administration Office documents. 
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Across years, the Institute has been consistent in maintaining high standards for funding only proposals to conduct 
rigorous research. From fiscal year (FY) 2002 to FY 08, the Institute’s education research funding rate (i.e., number of 
funded applications divided by the total number of responsive applications) has ranged from 7 percent to 12 percent 
(see table 4). From FY 06 to FY 08, the special education research funding rate has ranged from 9 percent to 13 percent 
(see table 5). Tables 4 and 5 also show the numbers of responsive applications and numbers of funded applications for 
each year. The numbers of responsive special education research applications and funded applications have fluctuated 
over the first 3 years of special education research competitions (table 5). The Board believes that with the establishment 
of long-term programs of research in special education that have annual competitions, the Institute will be able to grow 
its special education research programs. Although the funding rate of the education research program has been relatively 
stable, the increase in responsive applications has led to a notable increase in the number of funded applications. In 
2008, the Institute funded more than three times the number of education research applications that it funded in 2002 
while receiving about two and one-half times times the number of responsive applications (see table 4). 



Table 4 



Percentage of education research applications that were funded out of the 
total number of responsive applications 

Number of responsive Number of funded Funding rate 

Year applications applications (Percent) 



2002 


239 


21 


9 


2003 


421 


36 


9 


2004 


507 


36 


7 


2005 


479 


54 


11 


2006 


442 


37 


8 


2007 


484 


48 


10 


2008 


604 


70 


12 


NOTE: Percentages reflect the total number of NCER-funded applications for the education research competitions (including 
competitions for regular research programs and research and development centers, but excludes training grants and unsolicited 
grants) divided by the total number of responsive applications received for those competitions. 

SOURCE: lES Grants Administration Office documents. 



Table 5 



Percentage of special education research applications that were funded out of the 





total number of responsive applications 




Year 


Number of responsive 
applications 


Number of funded 
applications 


Funding rate 
(Percent) 


2006 


240 


27 


11 


2007 


294 


38 


13 


2008 


211 


20 


9 



NOTE: Percentages reflect the total number of funded applications for the special education research competitions (including 
competitions for regular research programs and research and development centers, but excludes training grants and unsolicited 
grants) divided by the total number of responsive applications received for those competitions. 

SOURCE: lES Grants Administration Office documents. 
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The Institute has made great strides in increasing interest in scientifically based research and in developing a strong 
education research portfolio. At the same time, the Board notes that the funding rate varies greatly by research 
program — the highest rates have been in the Cognition and Student Learning research program with an average 
funding rate over the last 2 years of 25 percent. The Cognition and Student Learning research program has attracted 
a number of well-trained and prominent researchers who have previously been funded through the National Science 
Foundation (NSF) and NIH. 

The Institute has also implemented a number of strategies to expand the involvement of qualified researchers in other 
areas. The capacity-building efforts include (a) actively recruiting strong researchers, with a history of funding through 
NSF or NUT, who have not previously sought funding from ED to apply to its programs; (b) providing technical 
assistance to applicants; (c) providing research training workshops to improve capacity of education researchers; and 
(d) without compromising the rigor of its research, being willing to fund proposals that have fixable, minor flaws. The 
Board believes that these efforts are important and should be continued. 

In light of concerns that the Institute has narrowly focused its research funding on projects that utilize randomized 
controlled trials (RCTs) to evaluate the effectiveness of interventions, the Board considered it important to examine 
the diversity of funded education and special education research projects and of the methodologies that are used to 
answer the research questions addressed in those projects. In FY 04, the Institute structured its research programs to 
reflect the types of projects that it would fund. Across the education and special education research programs, there 
are projects (a) to explore the relations between malleable factors (i.e., things that can be changed, such as student 
competencies and education practices) and education outcomes to identify potential targets of interventions; (b) to 
develop new education interventions; (c) to evaluate the efficacy of interventions under limited circumstances; (d) to 
evaluate the impact of interventions that are implemented at scale and under real world conditions; and (e) to develop 
and/or validate measurement instruments. (Appendix A provides a more detailed description of the types of projects.) 

In its funding announcements, the Institute states that projects intended to explore or identify relations between 
malleable factors and education outcomes typically involve correlational analyses of survey, observational, or 
administrative data. Development of new interventions incorporates a variety of methodologies — descriptive, 
observational, interview, qualitative, quasi-experimental, and experimental methodologies. Only for efficacy and 
scale-up evaluations does the Institute’s Request for Applications specify that randomized controlled experimental 
designs are preferred; in such cases, the Institute indicates that the research plans should include methodologies (e.g., 
observational, survey, qualitative) to examine factors that mediate and affect the relation between the intervention 
and its intended outcomes (i.e., studies that are often called “multiple methods studies”). The Board also noted that 
for efficacy and scale-up evaluations, the Institute indicates that it accepts evaluation designs other than random 
assignment studies, such as regression discontinuity designs, instrumental variables, and other well-designed quasi- 
experimental studies. The fifth type of project is conducting research to develop and validate measurement tools. 
According to the Institute’s funding announcements, such projects typically utilize correlational data to assess the 
reliability and validity of the instrument. 

In its funding announcements, the Institute clearly requests research projects that are diverse in purpose (e.g., 
developing interventions, exploring malleable factors) and in methodological requirements (e.g., correlational, 
descriptive, observational, qualitative, quasi-experimental, and experimental methodologies). 
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The Board examined the number of awards in each of the different research categories. As can be seen in table 6, about 
26 percent of the grants funded under the education research programs and 23 percent of the grants funded under the 
special education research programs are evaluations of the impact of interventions (efficacy and scale-up projects) for 
which the Institute prefers experimental designs. 



Table 6 



Center 

Education 

2004-08 



Numbers of grants by type of project 



Identify/Explore 


Develop 


Efficacy 


Scale-up 


Measurement 


21 


123 


53 


6 


28 



Special 

Education 4 49 18 1 11 

2006-08 



NOTE: Includes grants funded through the regular education research and special education research competitions. (Counts 
do not include grants awarded under competitions for which the Institute’s research goal structure did not apply (e.g., all grants 
awarded prior to 2004, all research and development center awards). 

SOURCE: lES Search Funded Research Grants tool. 



These data indicate that there are many misconceptions in the field regarding the types of projects and methodological 
approaches the Institute funds through its research grant programs. The Board congratulates the Institute for 
developing methodologically rigorous and diverse research portfolios in its first 6 years of operation. 

A.l.b. NCEE Evaluation Studies 

Whereas the Institute’s research grant programs address a wide range of education research questions and utilize a 
broad range of methodological approaches, NCEE’s evaluation division primarily focuses on questions of what works, 
for whom, and under what circumstances. NCEE evaluates the impact of programs administered by ED, using 
methodologically rigorous designs applied to large samples of students and schools. The selection of what will be 
evaluated in NCEE studies is determined by Congressional requests and ED’s priorities for evaluations of its federal 
programs. To date, 26 large evaluations have been completed or are currently being conducted. (See appendix B for 
a list of studies and their design.) Twenty-two of these 26 evaluations use randomized controlled designs to test the 
comparison between the treatment intervention(s) and the control condition. 

In contrast, consider previous evaluations contracted by ED. Robert Boruch of the University of Pennsylvania and 
his colleagues examined 144 contracts awarded by ED’s Office of Planning and Evaluation Service for evaluation 
studies during 1995-97. Fifty-one evaluations were identified as addressing the impact of federal programs, but only 
five of these used a randomized controlled design to measure that impact. Similarly, in their review of the 84 program 
evaluations and studies planned by ED for FY 2000, they found 16 evaluations that addressed the impact of federal 
programs, but only one involved a randomized field trial to assess the impact.'* The Institute should be applauded for 
its efforts to greatly improve the rigor of impact evaluations of ED programs. 



^ Boruch, R., DeMoya, D., and Snyder, B. (2001). The Importance of Randomized Field Trials in Education and Related Areas. In R. Boruch 
and F. Mosteller (Eds.), Evidence Matters: Randomized Trials in Education. Washington, DC: Brookings Institution Press. 
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In addition to implementing impact evaluations that are appropriately designed to determine the causal influence 
of interventions, the Board commends the Institute for implementing a process by which knowledge garnered from 
initial evaluations of interventions is utilized to inform the development or improvement of interventions that are 
subsequently evaluated. Rigorous evaluations of federal education programs enable ED to determine which programs 
are improving student outcomes as well as to identify programs that need modification or improvement to achieve 
their desired objectives. 

An important example comes from NCEE’s initial and follow-up evaluations of afterschool programs. In October 
2004, NCEE released its first report on an evaluation of afterschool programs.^ In 1999, Mathematica Policy Research, 
Inc., and Decision Information Resources, Inc., under contract to ED, launched two evaluation studies of the 21st 
Century Community Learning Centers program. In the first study, the outcomes of elementary school students who 
were randomly assigned to the program were compared to the outcomes of students who were not. The second study 
used a matched-comparison design in which middle-school students who participated in the program were compared 
to a group of similar students who did not. In general, there were few differences between the students who were in the 
intervention group and those who were not. There were few impacts of the program on academic achievement, and 
there was no difference between the treatment and control (elementary school) or comparison (middle school) groups 
in receiving homework assistance, tfowever, access to the afterschool program resulted in elementary students in the 
treatment group reporting that they felt safer than elementary students in the control group. The lack of impact on 
academic outcomes was a disappointing, but not uncommon, result for scientific studies. These findings, nonetheless, 
provided critical information for improving the design of afterschool programs. Across the programs, researchers 
noted that academic support was primarily limited to time for students to work on homework; relatively few sessions 
provided additional instruction (e.g., assistance with homework, math instruction). 

Based on these findings, the Institute supported the development and evaluation of a math curriculum and a 
reading curriculum that were designed to be used in afterschool programs with elementary school children who are 
behind grade level.® Would participation in afterschool programs that are specifically designed to increase academic 
instruction improve academic outcomes? For the evaluation, students were randomly assigned to receive the enhanced 
afterschool program or the regularly available program. In June 2008, the Institute reported the flrst-year findings of 
the evaluation. Relative to students in the regular afterschool program group, students in the enhanced math program 
received 30 percent more hours of math instruction over the school year and attained 8.5 percent more growth 
over the school year as measured by the SAT 10 total math score. For the reading program, however, there were no 
significant impacts on the SAT 10 reading tests (total and subtests). The evaluation is ongoing, with a second year of 
program implementation and data collection to determine whether there may be a cumulative effect of the enhanced 
afterschool program on continuing students and to assess the impact of a more mature program on new students. 

The combination of these two evaluations (the initial one with null findings and the most recent one) demonstrates 
how important rigorous impact evaluations are to the improvement of existing programs. The Board commends 
the Institute for initiating a process that uses the findings of an evaluation to feed into the modification of existing 
programs in order to improve outcomes. 



^ Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., and Mansfield, W. (2004). When Schoob Stay Open Late: The National 
Evaluation of the 21st Century Community Learning Centers Program: New Findings. National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U.S. Department of Education. Washington, DC. 

^ Participants in the study were limited to students in grades 2 through 5 who (a) were behind grade level, but no more than 2 years behind grade 
level; (b) did not have severe learning disabilities or behavioral problems; and (c) were able to receive instruction in English. Source: Black, 

A.R., Doolittle, E, Zhu, E, Unterman, R., and Grossman, J.B. (2008). The Evaluation of Enhanced Academic Instruction in Afierschool Programs: 
Findings After the First Year of Implementation (NCEE 2008-4021). National Center for Education Evaluation and Regional Assistance, Institute 
of Education Sciences, U.S. Department of Education. Washington, DC. 
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This past year, the Institute initiated an overall review of its evaluation program, bringing in three eminent researchers 
with decades of experience conducting rigorous evaluations of education and social interventions to review the 
evaluation projects contracted through NCEE. The Board is impressed that the Institute is taking the initiative to 
improve its evaluation program. The Board commends the Institute for the progress it has made thus far and looks 
forward to continued advances based on this external review. 

A.I.C. Regional Educational Laboratories 

Congress created the REE program in 1965 to “conduct long-term activities to address national educational 
problems.” Various reports on the RELs criticized the overall quality of the research activities they conducted, leading 
to a significant shift under ESRA. The Institute was required to overhaul the REE program and bring it up to the 
Institute’s standards for rigorous and relevant research. NCEE undertook this task when the 10 new contracts for RELs 
were awarded in 2006. The emphasis on having the RELs provide scientifically valid information resulted in a number 
of significant changes to the program. The RELs are now responsible for providing technical assistance to policymakers 
and practitioners in interpreting the meaning of and applying scientifically based research findings, conducting 
research to quickly respond to needs expressed by educators and policymakers in their region, and conducting rigorous 
education research and evaluation. Most significantly, ESRA required that a peer review process be established to 
ensure that all REE research and evaluation projects and products meet the Institute’s standards for rigorous research.^ 

The role of the RELs was thus redefined to require that they operate as intermediaries between the world of research 
and the world of practitioners. Rather than focusing on professional development, the RELs would provide technical 
assistance through applied research and development projects on issues of importance within the regions. For example, 
the RELs are currently conducting 25 large-scale RCTs. (See appendix C for a list.) The questions selected for these 
studies relate to high-priority issues in the region; many studies are testing a state initiative (program or intervention) 
to determine whether it is producing the desired impact. For example, REE Southeast is evaluating the Alabama 
Math, Science, and Technology Initiative developed by the Alabama State Department of Education to improve the 
quality of mathematics and science instruction. The evaluation will determine whether the program increases student 
achievement in math, science, and reading and enhances teacher instructional practices. In response to California and 
other western states making high school economics coursetaking a graduation requirement, REE West implemented an 
evaluation of Problem-Based Economics, a high school curriculum designed to increase class participation and content 
knowledge, especially for low-achieving students. 

Another change for the REE program is the provision of technical assistance through fast-response projects. These 
projects are on current education issues of importance at local, state, and regional levels, and are identified through 
the RELs’ outreach efforts and requests for assistance from state and local educators and policymakers. Fast-response 
projects result in peer-reviewed reports that are available through two online series — Issues & Answers and REL 
Technical Briefs. By September 2008, the Institute had released 61 of these typically descriptive reports, such as a report 
for Utah on the implementation of its K through 3 program, and a report that summarized state standards for high 
schools in the Central Region states. 

All REL reports and activities can be accessed online at http://ies.ed.gov/ncee/edlabs/ , the central website. RELs are 
expected to broadly disseminate these reports and other materials that demonstrate the hallmarks of trustworthy 
evidence, and help practitioners recognize and distill conclusions from high-quality evidence. 



^ The standards for the scientific peer review process are posted on the Institute’s website, as are REL reports that meet the Institute’s standards 
and pass peer review ( http://ies.ed.gov/ncee/edlabs ). 
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The Board is pleased with the Institute’s progress in establishing processes to improve the quality of the research 
conducted through the REL program. Given the changes in the focus of REL activities, however, the Board is 
interested in state and district perspectives on such changes. Would states and districts, for example, find it more useful 
for the RELs to put greater emphasis on conducting rigorous fast-response studies and less emphasis on randomized 
trials, or are the RCTs a welcomed innovation? The Board wonders if the RELs would better serve their regions if they 
were primarily generators of locally relevant knowledge. As more states establish statewide longitudinal databases, the 
Institute might consider expanding the number of REL projects that carry out policy- relevant research using databases 
from states and districts in their regions, similar to the type of research conducted by the Consortium on Chicago 
School Research. Such research would be more evaluative in nature than the typical fast-response projects that the 
RELs currently conduct and be intended to inform policy revision or modification. For example, these projects could 
incorporate quasi-experimental designs (e.g., comparative interrupted time series) using primarily administrative data 
and, although correlational in nature, would go beyond simple counts of those states that are doing X and Y or case 
studies of programs operated in a region. As the RELs move toward completing their 5-year contracts, the Board 
recommends that the Institute conduct an evaluation of the RELs, including feedback from state and local education 
policymakers and decisionmakers on the REL activities and reports. 

A.2. Is the Institute increasing the number of scientists from a variety of discipiines 
who have entered the fieid of education research and are activeiy invoived 
in conducting high-quaiity scientific research in education? is the Institute 
improving the nation’s capacity to conduct rigorous education research by 
advancing the methodoiogicai and statisticai knowiedge and toois avaiiabie to 
education scientists? 

To determine the extent to which the Institute is building capacity in the field to conduct more rigorous research, 
the Board examined the Institute’s predoctoral and postdoctoral research training programs and its research training 
workshops. Information about these programs is provided below. Based on its review of these programs, the Board 
concludes that over the past 6 years the Institute has succeeded in producing new education scientists with the 
expertise and skills to conduct rigorous education research and enhance the knowledge and skills of current researchers 
to conduct rigorous studies. 

A.2.a. Predoctoral and Postdoctoral Research Training Programs 

The Institute initiated its predoctoral and postdoctoral research training programs in the education sciences to 
increase the supply of scientists who are prepared to conduct rigorous education research, including developing new 
interventions, carrying out rigorous evaluations of education programs and policies, and designing and validating 
assessments and measurement tools. In FY 04 and FY 05, initial competitions were held for the predoctoral research 
training programs; 10 institutions received awards to establish interdisciplinary programs in the education sciences. 

(See appendix D for a list of institutions.) 

From 2004 through 2008, 233 predoctoral fellows have been or are being trained. The quality of the students 
recruited into these training programs has been high. The average CUE scores among the 233 predoctoral fellows are 
Verbal 626 and Quantitative 704.® For comparison purposes, the mean GRE scores for all students nationally who 



® All information on predoctoral and postdoctoral research training programs from: lES/NCER 2004-2008 Research Training Programs in the 
Education Sciences Report. 
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were tested between July 1, 2003, and June 30, 2006, for students intending graduate studies in the social sciences, 
were Verbal 487 and Quantitative 563, and for students intending graduate studies in education. Verbal 449 and 
Quantitative 5337 According to U.S. News & World Report^ the average GRE scores for incoming doctoral students 
to education programs that are ranked in the top 25 education graduate programs are Verbal 563 and Quantitative 
642. The predoctoral fellows funded by the Institute are clearly among the best students pursuing doctoral training in 
education research. 

As of the summer of 2008, 39 predoctoral fellows had completed their doctoral programs and obtained employment. 
Twenty-seven (69 percent) of the 39 completed predoctoral fellows are engaged in education research in their 
postfellowship employment; an additional 3 fellows (8 percent) are pursuing both education research and other types 
of research. Six (15 percent) are engaged in research but not education research, and 3 (8 percent) are not doing 
research at all in their postfellowship employment. Thus, a total of 30 (77 percent) of completed predoctoral fellows 
are engaged in education research. (Appendix E reports numbers of publications and refereed conference presentations 
for predoctoral fellows.) 

In FY 05, the Institute began competitions for postdoctoral education research training programs; 13 postdoctoral 
research training programs were awarded between FY 05 and FY 07. The postdoctoral training programs are smaller 
by design than the predoctoral programs; each postdoctoral program trains up to four fellows. From fall 2005 through 
June 2008, 30 postdoctoral fellows have been or are being trained. 

As of the summer of 2008, 12 postdoctoral fellows have obtained postfellowship positions of employment. Eight 
(67 percent) of the 12 completed postdoctoral fellows are engaged in education research in their postfellowship 
employment; an additional 2(17 percent) are pursuing both education research and other types of research. Two (17 
percent) completed postdoctoral fellows are engaged in research but not education research. Thus, a total of 10 (84 
percent) of completed postdoctoral fellows have begun careers in education research. (Appendix E reports numbers of 
publications and refereed conference presentations for postdoctoral fellows.) 

The Board finds the Institute’s predoctoral and postdoctoral research programs to be highly successful at training a 
new generation of education researchers. In less than 4 years, these programs have launched 40 new and well-trained 
education researchers." The Board is particularly pleased to report that one postdoctoral fellow who begins as an 
assistant professor in the fall of 2008 has already received funding from the Institute to carry out an early childhood 
research project. The Board recommends that the Institute continue to invest in predoctoral and postdoctoral research 
training. 

A.2.b. Research Training Workshops 

In addition to supporting the preparation of new education scientists, the Institute has conducted a number of 
research training workshops. In the summers of 2007 and 2008, the Institute held a 2-week intensive research training 



’ GRE Guide to the Use of Scores 2007-2008. Retrieved June 30, 2008, from http://www.ets.org/Media/Tests/GRE/pdf/994994.pdf 
U.S. News & World Report. (2008, December). 2009 Edition: U.S. News & World Report’s Americas Best Graduate Schools. Washington, DC: 
U.S. News & World Report LE 

" The Board also examined attrition among the predoctoral fellows and found it to be low. Between fall 2004 and June 2008, 15 (6.4 percent) 
predoctoral fellows left their training programs without completing a Ph.D. Among the postdoctoral fellows, as of June 2008, 3 fellows (10 
percent) terminated their fellowships either during or immediately following their first year. 
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workshop on the design, implementation, and analysis of cluster randomized trials to increase the national capacity of 
researchers to develop and conduct rigorous evaluations of the impact of education interventions. The instructors for 
these two training workshops were premier evaluation methodologists and statisticians. Thirty education researchers 
participated in each of the summer workshops. Because relatively little education research included experimental 
evaluations of interventions in the latter half of the 20th century, existing capacity to conduct rigorous evaluations of 
the effect of education interventions on intended outcomes is limited. These workshops are an important complement 
to the predoctoral and postdoctoral research training programs because they focus on improving the skills of practicing 
researchers. As of August 2008, of the 60 participants in the 2 summer workshops, 42 researchers are currently 
working on research grants or contracts supported by the Institute or have submitted grant applications to the 
Institute. 

Through all of its special education research programs, the Institute encourages research on programs and practices 
for improving educational and developmental outcomes for individuals with low-incidence disabilities (e.g., visual or 
hearing impairments). To enhance capacity for research on low-incidence disabilities, the Institute sponsored a 2-day 
workshop in April 2008, to provide researchers with specialized training on using single-case methodologies that 
incorporate quantitative analyses. Single-case methodologies are experimental approaches for determining the effect of 
interventions on individuals and are used primarily with low-incidence disabilities. Thirty-nine researchers from across 
the country participated, including relatively new researchers to very senior people in the field. About one-third of 
the researchers reported having grants from the Institute. Eleven of the participants are currently or have been editors 
or associate editors of major research journals, and more than half are or have been on the editorial boards of major 
research journals. Through this workshop, the Institute has been able to reach many researchers who have influence 
over the field. 

The Institute continues to provide workshops to enable researchers to conduct analyses of data from its surveys. In 
FY 08, NCES held 9 such 2- to 3-day workshops that covered 15 surveys, including, for example, the Birth and 
Kindergarten cohorts of the Early Childhood Longitudinal Study, the Education Longitudinal Study of 2002, the 
Schools and Staffing Survey, the National Assessment of Adult Literacy, and a set of international studies. The 
workshops are intended to familiarize participants with NCES survey data. The general format is constant across 
workshops. Each one starts with an overview of the content and design of the featured survey, followed by an overview 
of technical and methodological issues associated with the use of that survey. Then, after receiving instruction on how 
to access and analyze the data, participants conduct their own analysis of the data. In many of the workshops, the 
participants are encouraged to formulate potential research questions, and assistance is provided in developing and 
analyzing the research question. During FY 08, a total of 196 potential data users participated in the NCES survey 
training workshops. 

The members of the Board are impressed with the variety of efforts the Institute has launched to increase education 
research capacity in our nation. The Board believes that the research training programs designed to enhance the 
skills of existing researchers are critical for enabling the country to quickly progress toward improving the quality 
of education in our schools. The Board recommends that the Institute continue to invest in these research-training 
programs. 



’ See the lES website, Calendar of Events, at http://ies.ed.gov/whatsnew/calendar/?tid=.3&cid=2tos=9-2007-12 | m . 
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A. 2.C. Advancing Methodological and Statistical Tools for Education Scientists 

Finally, the Institute has launched two efforts to improve the scientific tools available to education researchers. First, 
through its Methods Group, NCEE brings together experts in evaluation methodologies and statistics. Critical issues 
related to improving evaluation methods and statistical analyses are identified. NCEE supports investigations to 
resolve and advance methods. The reports from this group provide guidance for the design, implementation, and 
analysis of future evaluation projects contracted by NCEE and for researchers funded through NCER and NCSER. 

For example, the group examined the problem of “false discoveries,” and their discussions helped NCEE produce a 
guide for the field. Second, the Institute recently initiated a research program through NCER to fund research to 
improve education research methods and statistical analyses. The Board believes that it is critical to study and improve 
a wide range of methodologies that are available to education researchers (e.g., value-added methods, single-subject 
experimental designs). Science is only as good as its research methods. The Board is pleased that the Institute is seeking 
to make advances in this arena. 

B. Increasing the Relevance and Usefulness of Education Research 

B. 1 . Is the Institute increasing the relevance of the research it funds or conducts in 
order to meet the needs of the Institute’s customers? 

In 2002, the Institute conducted a survey of education policymakers to obtain input on the issues and questions they 
wanted researchers to address and to gain their general impressions of education research. Their general sentiments 
on education research are reflected in the comments of one veteran superintendent: 

There may be less than one percent of the existing research that’s really meaningful to teachers. Much 
is for researchers, for getting funding, for career advancement, or for advocacy....! don’t want theories. 

Teachers need strategies, practices. Give them things that can help teaching and learning, things that 
can help kids. 

In response to input from education leaders and decisionmakers, the Institute sought to create research, statistics, 
and evaluation programs that are explicitly intended to serve the needs of those who are responsible for the education 
of students in our country. To examine the extent to which and the ways in which the Institute has been successful 
in increasing relevance of education research, the Board considered data on the relevance of education and special 
education research programs, the work of the National Research and Development Centers, the timeliness and 
relevance of NCES reports, and the timeliness and relevance of NCEE evaluations. 

B.l.a. Relevance of the Education and Special Education Research Programs 

Since its inception, the Institute has been committed to conducting research that meets the needs of the education 
practice community. The Institute has actively reached out to education leaders and decisionmakers through a number 
of venues. In 2002, the Institute conducted a survey of education policymakers, including superintendents, chief 
state school officers, state higher education executive officers, state legislators, governors’ education policy advisors. 



Schochet, P.Z. (2008). Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations (NCEE 2008-4018). National Center for 
Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Washington, D.C. This peer- 
reviewed report is part of a series called Technical Methods Reports that is available online at http://ies.ed.gov/ncee/pubs . 

Institute of Education Sciences: Eindings Erom Interviews With Education Policymakers. (2002). Retrieved from www.eric.ed.gov . (ERIC 
ED4801). 
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congressional staff members, and education association executive directors. According to the respondents of this 
survey, the highest priority issues in need of further research were teacher quality, standards-based assessment and 
accountability, education finance-related issues, curriculum and instruction, and closing achievement gaps. 

In November 2006, the Institute convened the Urban Education Research Task Force, comprised of leaders of large 
urban school districts, representatives from the Council of the Great City Schools, a chief state school officer, and 
researchers who focus on urban education. The Task Force has provided the Institute with timely input on the research 
needs of large urban school districts. 

Input from education decisionmakers has informed the development of the Institute’s education research programs. 
The Institute has developed a set of long-term programs of research that focus on improving education practice in 
those areas that relate to the Institute’s priorities (e.g., Reading and Writing, Middle and High School Reform, Teacher 
Quality). Within each research program, the Institute encourages a broad range of applications. For example, in 
Mathematics and Science, applicants can propose to explore the relations between malleable factors and mathematics 
or science outcomes for students from kindergarten through grade 12 or to develop or evaluate (a) mathematics 
or science curricula or instructional approaches for students at any level from kindergarten through grade 12, (b) 
curricula or instructional approaches for teaching basic mathematics skills to adults through adult and vocational 
education programs or through developmental/bridge programs designed to help underprepared students acquire 
the skills to succeed in college, or (c) mathematics or science assessments to support instruction from kindergarten 
through high school or to support teaching basic mathematics skills to adult. The Institute’s focused programs of 
research allow the Institute to accept a wide range of projects while providing some constraints to keep projects 
focused on generating solutions to the issues that are of greatest need to education leaders and practitioners. 

To evaluate the relevance of the projects the Institute has funded, an external panel of experienced education leaders 
and practitioners evaluates a randomly selected sample of newly funded research projects for overall relevance of the 
proposed research to education in our country. Panel members consider the degree to which the proposed research 
addresses problems that are of national significance to education and have the potential to contribute to solving those 
problems. Relevance is rated on a 9-point Fikert-type scale where 1 represents “Very low relevance” and 9 represents 
“Very high relevance.” 

Table 7 reports the average relevance ratings beginning in FY 01 with ratings for projects awarded through the last 
Field Initiated Studies competition held by the Institute’s predecessor, OERI. As indicated in table 7, the average 
relevance ratings for the Institute’s education research projects have been in the adequate-to-high relevance range; 
the rating for the last year of projects funded under OERI was adequate relevance. As shown in table 8, the special 
education research projects have been rated as being of high relevance. 
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Table 7 



Relevance of education research grants 



Fiscal year 


Mean rating* 


Standard deviation 


2001 OERI** 


5.5 


1.1 


2002 NCER 


6.0 


1.2 


2003 NCER 


6.5 


.9 


2004 NCER 


6.1 


.8 


2005 NCER 


5.9 


.9 


2006 NCER 


6.9 


.7 


2007 NCER 


6.5 


.9 



*On a rating scale of 1 to 9, 5 = “Adequate relevance”; 7 = “High relevance” 

**FY 01 was the last round of research projects funded under the Field Initiated Studies research 
program of OERI. 

SOURCE: lES Standards and Review Office. 



Table 8 



Relevance of special education research grants 



Fiscal year 


Mean rating* 


Standard deviation 


2006 NCSER 


6.6 


.6 


2007 NCSER 


6.9 


.8 



*On a rating scale of 1 to 9, 5 = “Adequate relevance”; 7 = “High relevance” 
SOURCE: lES Standards and Review Office. 



The Institute has undertaken a number of strategies to improve the relevance of education research and to ensure that 
the work that it funds addresses the needs of education leaders and practitioners. The Board’s evaluation is that the 
Institute has made substantial progress in improving the relevance of federally funded education research. The Board 
recommends that the Institute continue to seek input from education decisionmakers on the issues for which they 
need answers from education researchers. The Board commends the Institute for having education leaders (e.g., district 
superintendents and assistant superintendents, principals, program managers, and directors of state departments of 
education) rate the relevance of its research grants. 

B.l.b. National Research and Development Centers 

Through its National Research and Development Center program, the Institute funds research to develop and test 
solutions to specific education problems, as well as national leadership and dissemination activities. The Institute has 
used the Centers to generate relevant evidence in key policy areas (e.g., teacher incentives and disincentives, teacher 
credentials, assessment of English language learners). The Board recognizes here some of the innovative work that has 
been conducted by the research and development centers and is pleased with the direction and practical focus of the 
research and dissemination work of the National Research and Development Centers. 
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One of the most productive and innovative centers is the National Center for Analysis of Longitudinal Data in 
Education Research (CALDER). Using existing longitudinal state and district administrative databases from Florida, 
Missouri, New York, North Carolina, Texas, and Washington, CALDER researchers examine how state and local 
teacher policies on hiring, compensation, and certification, and state and local governance policies on accountability 
and choice, relate to student outcomes. In addition to generating new knowledge,'^ CALDER has sponsored two 
public conferences — one on the development and use of state and district administrative databases for research 
purposes, the other on the use of value-added measures in education policy contexts. More than 100 policymakers, 
educators, and researchers attended each of these meetings.'^ 

The National Center on Performance Incentives (NCPI) focuses on the concept of providing financial rewards 
to classroom teachers who significantly elevate students’ academic achievement, as an alternative to uniform salary 
schedules applicable to virtually all teachers. In addition to its major project (a large randomized controlled evaluation 
of the effects of student achievement-related bonuses for teachers on students’ mathematics achievement), NCPI 
conducts studies of teacher recruitment and the cost-effectiveness of various pay-for-performance strategies. More than 
300 policymakers, educators, journalists, and researchers attended NCPI’s first national conference in February 2008. 
NCPI also disseminates reports, journal publications, and links to performance-incentive projects throughout the 
nation on its website ( http://www.performanceincentives.org ). 

The Center for Research on Evaluation, Standards and Student Testing (CRESST) has focused part of its efforts on 
assisting states in validly assessing the academic performance of English language learners. CRESST recently released 
three major reports on assessment of English language learners: (a) Issues in Assessing English Language Learners: English 
Language Development Measures and Accommodation Uses — Literature Review (Volume I)\ (b) Issues in Assessing English 
Language Learners: English Language Development Measures and Accommodation Uses — State Practice Review (Volume 
Iiy, and (c) English Language Proficiency Assessment in the Nation. These papers, along with related external and internal 
research, are available at http://www.cse.ucla.edu . 

B.2. To what extent is the Institute producing findings and data in a timeiy manner 
that ensures their reievance to current and/or pressing education issues? 

Improving the scientific quality of education research and evaluation is a necessary step towards creating a research 
and evaluation enterprise that will serve the needs of education policymakers and practitioners. However, to inform 
education decisionmaking, the Institute must produce and release its major statistics and evaluation reports in a 
timely manner. The Board examined data on the timeliness and perceptions of relevance of the Institute’s statistics and 
evaluation reports. 

B.2.a. Timeliness and Relevance of NCES Reports 

NCES has a responsibility to get its data into the hands of its users as quickly as possible. To speed up the 
dissemination of data, in 2006 NCES shifted to a new release publication format, called Eirst Look. Previously, 



In less than 2 years, CALDER has released 18 working papers, 1 research note, and 1 policy brief (available at http://www.caldercenter.org/ ). 
Several papers examine policies designed to improve teacher qualifications in high-poverty schools. For example, one study found that Teach For 
America (TFA) teachers tend to have a positive effect on high school student test scores relative to non-TFA teachers, including those who are 
certified in-field. Such effects exceed the impact of additional years of experience and are particularly strong in mathematics and science. This 
paper was recently reviewed by the WWC and was evaluated as being consistent with WWC evidence standards with reservations. 

As an aside, the Board notes that CALDER’s research program — all secondary data analyses — attests to the Institute’s support of 
nonexperimental research. 
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data releases were accompanied by a more complicated publication that took more time to write and review. The 
streamlined First Look was a procedural change that wrought changes in the culture of NCES by emphasizing the 
importance of quick data releases. The timeliness data demonstrate the effectiveness of this change. Targets were 
established for decreasing the time between the end of data collection and the release of the first publication from 
that data collection. For FY 06, the Institute determined to issue the first release publications of NCES data within 
18 months of the end of data collection; for FY 07, the target was 16 months. For the purpose of comparison, the 
Institute used data from its FY 05 NCES publications as baseline data.'^ 

For the baseline data, the average time to the first release publication was 19.8 months. In FY 06, the Institute reduced 
the average time to the first release publication to 14.4 months. In FY 07, the average time to the first release NCES 
publication was 12.3 months. 

Perhaps the most important statistics reports that the Institute produces are the reports of the National Assessment 
of Educational Progress (NAEP). Data for the reading and mathematics assessments are collected every 2 years. The 
Institute established a goal of releasing the initial report of NAEP results within 6 months of the completion of data 
collection. In 2003 (baseline year), the initial release of results was 8 months after data collection was completed. In 
2005, the report was out in 6 months; in 2007, the report was out in 5.25 months. 

The Institute has done a tremendous job in reducing the time to publication of its NCES reports. To accomplish such 
improvement in a relatively short period of time is an impressive feat for which the Institute can be proud. 

The Institute also collects customer satisfaction data on a number of indices, including customer satisfaction with 
the timeliness of NCES publications. Data for 1997 through 2004 were collected through a random sample of more 
than 3,900 academic researchers, education associations, education journalists, users of NCES’s National Education 
Data Resource Center, and federal, state, and local policymakers. Table 9 below shows the percentage of customer 
respondents satisfied or very satisfied with the timeliness of NCES publications. As can be seen, in 2007 the majority 
(86 percent) of customers were satisfied or very satisfied with the timeliness of NCES publications. 



Table 9 



Percentage of customer respondents satisfied or 
very satisfied with the timeliness of 
NCES publications 

Year Actual 



1997 


72 




1999 


77 




2001 


74 




2004 


78 




2006 


85 




2007 


86 




SOURCE: National Center for Education Statistics. 



'^The baseline data included some data from years prior to 2005 to capture comparison data for the most recent corresponding major NCES 
releases. 
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In addition, the Institute collects customer service survey data on satisfaction with the timeliness of NCES data files. 
As can be seen in table 10 below, the percentage of customers surveyed who report being satisfied or very satisfied 
with the timeliness of NCES data files has risen over the last 10 years. The vast majority (more than 80 percent) of 
customers surveyed in 2006 and 2007 were satisfied or very satisfied with the timeliness of NCES data files. 



Table 10 


Percentage of customer respondents satisfied or 
very satisfied with the timeliness of NCES datafiles 


Year 




Percent 


1997 




52 


1999 




67 


2001 




66 


2004 




78 


2006 




86 


2007 




84 


SOURCE: National Center for Education Statistics. 



In 1997, 1999, 2001, and 2004, NCES administered a customer survey to a random sample of more than 3,900 
federal, state, and local policymakers, academic researchers, education association researchers, education journalists, 
and known NCES users. Across these years, approximately 90 percent of respondents reported being satisfied or 
very satisfied with the relevance of NCES publications and services. In 2004, NCES included representatives from 
additional groups in its customer survey. Table 1 1 shows the percentage of respondents who reported being satisfied or 
very satisfied with the relevance of NCES publications by group. More than 90 percent of the policymakers, education 
administrators, and teachers were satisfied or very satisfied with the relevance of NCES publications. Only one 
category — media — fell below 90 percent. 



Table 1 1 



Percentage of NCES customer survey respondents satisfied or 
very satisfied with the relevance of NCES publications in 2004 



All types Policymakers 


Administrators, 
supervisors, 
or managers 


Teachers 


Researchers 


Reporters/ 

media 


Other 


93 93 


91 


98 


93 


76 


92 


SOURCE: National Center for Education Statistics, 2004 Customer Survey. 



Overall, the Board is impressed with the timeliness and relevance of NCES publications. The perception of the 
relevance of NCES reports has historically been high and that has been maintained. The major accomplishment during 
the past 6 years has been the marked improvement in the timeliness of NCES reports. 
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B.2.b. Timeliness and Relevance of NCEE Evaluation Reports 

Since 2007, the average number of weeks from submission of a report to the Standards and Review Office to approval 
of the report is approximately 28 weeks.'® On average, that time was divided between 13 weeks for the Standards and 
Review Office to obtain external reviews and to review the initial version and subsequent revisions of the report and 
14 weeks for NCEE and its contractor to make revisions to the report. In addition, the number of weeks that it takes 
to prepare the final publication version of the report (i.e., weeks from peer review approval to release of report) is 8 
weeks. For the 8 reports that had been released at the time that the Board’s report was being written, the average total 
elapsed time'® from submission of a report to the Standards and Review Office to release of the report is 34 weeks 
(approximately 8.5 months) with a range of 13 weeks to 47 weeks. Five of the 8 reports were released within the 
10-month goal. 

At the time that the Board requested these data,^° five reports had entered the peer review process but had not yet been 
approved. The average number of weeks that these reports have been in the review process is approximately 25 weeks 
with 1 1 weeks in the Standards and Review Office and 14 weeks in the hands of NCEE and its contractor. 

The Board recognizes the challenges in producing complex and lengthy evaluation reports in a timely manner and 
appreciates the Institute’s efforts to date to reduce this time. However, the Board believes that the average number 
of weeks from contractor submission of an initial draft to having a report approved and released should be reduced 
further. 

In addition to the strategies the Institute is currently employing to reduce the time to release of reports, the Board 
recommends that the Standards and Review Office prepare a document that details the Institute’s standards and that 
this document be available to all those who are preparing reports. The Board also strongly recommends that NCEE 
establish procedures to ensure timely receipt of reports and revisions of reports from its contractors and to ensure 
efficient handling of drafts while they are in the hands of NCEE staff. 

To evaluate the relevance of NCEE projects, the Institute had an external panel of experienced education leaders and 
practitioners evaluate the relevance of the plans for NCEE evaluations that were launched in FY 04 through FY 06.^' 
The panel members were asked to rate the overall relevance of the evaluation project to education in our country. They 
were asked to consider the degree to which the evaluation project addresses problems that are of national significance 
to education and have the potential to contribute to solving those problems. Relevance is rated on a 9-point scale 
where 1 represents “Very low relevance” and 9 represents “Very high relevance.” These ratings are similar to the ratings 
of the education research and special education research proposals. Table 12 reports the average relevance ratings of 
the NCEE evaluations and includes for the sake of comparison, the ratings for projects awarded through the last Field 
Initiated Studies competition held by the Institute’s predecessor, OERI. As indicated in table 12, the average relevance 
ratings for the Institute’s evaluation projects have been in the adequate-to-high relevance range; the rating for the last 
year of projects funded under the OERI was adequate relevance. 



Based on data from 10 reports. 

Total elapsed time is the number of weeks from submission of report to peer review to the release of the report. This includes weekends and 
holidays. 

“ September 5, 2008. 

These relevance data are ratings of the relevance of the project plans and were obtained prior to the execution and completion of the 
evaluations. 
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Table 12 



Relevance of plans for NCEE evaluation projects 



Fiscal year 


Mean rating* 


Standard deviation 


2001 OERF* 


5.5 


1.1 


2004 NCEE 


7.2 


.9 


2005 NCEE 


6.8 


.6 


2006 NCEE 


5.9 


.7 



*On a rating scale of 1 to 9, 5 = “Adequate relevance”; 7 = “High relevance” 
**FY 01 was the last round of research projects funded under the Field 
Initiated Studies research program of OERl. 

SOURCE: lES Standards and Review Office. 



On average, NCEE evaluations were rated as being adequately to highly relevant projects. The Board commends the 
Institute for implementing evaluations that are useful for improving education in our country. The Board, however, 
notes that the Institute no longer collects ratings of the relevance of the plans for NCEE evaluations and, perhaps 
more importantly, does not at this point routinely collect data on users’ perceptions of the relevance of NCEE reports. 
The Board recommends that the Institute establish a regular process for obtaining relevance data on NCEE evaluation 
reports. One such area is helping policymakers and practitioners to critically evaluate claims regarding the implications 
of research studies. 

C. Facilitating Evidence-Based Decisionmaking 

Facilitating evidence-based decisionmaking is a multistep process. The Institute must first produce information that 
can be used to inform education practice. Then the Institute must widely disseminate the information in ways that 
allow education decisionmakers and practitioners to use the information. Finally, although the Board recognizes that 
responsibility for providing technical assistance to education policymakers and decisionmakers, as well as to state and 
local education agencies, is spread across many units within ED, the Institute bears some responsibility for providing 
general technical assistance through the RELs. In addition, the Board believes that the Institute has responsibility for 
providing technical assistance in areas for which the Institute has unique capabilities for providing expert guidance. 

C. 1 . Is the Institute developing or identifying a substantial number of programs, 

practices, policies, and approaches that enhance academic achievement and 
that can be widely deployed? Is the Institute identifying what does not work 
and what is problematic or inefficient, and thereby encouraging innovation and 
further research? 

To be successful, the Institute must produce information that can be used to inform the decisions of education 
decisionmakers. In a relatively short period of time, the Institute has produced scientifically based evidence about 
education programs, practices, policies, and approaches that can be utilized by education decisionmakers and 
practitioners. 
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C.l.a. What Works Clearinghouse/Intervention Reports 

One of the chief accomplishments of the Institute is the creation of the WWC, which evaluates the evidence on the 
effectiveness of education programs, practices, and policies and makes its reports available through its website at 
http://www.whatworks.ed.gov . The WWC utilizes transparent, rule-based evidence standards, rating schemes, and a 
reporting system. Because there have been questions about the scientific validity of the WWC review process and the 
nature of the intervention reports, the Board commissioned an independent review of WWC procedures by a panel of 
experts in meta-analysis and program evaluation. The panel concluded that: 

Overall, the panel believes that the WWC review and processes are based on scientifically appropriate 
methodologies for the task of judging the strength of the evidence regarding the effectiveness of the 
interventions identified in the topic areas, although the panel did not have time or resources to fully 
investigate the application of these methodologies in every review. Moreover, the panel believes that the 
Intervention and Topic Area Reports provide succinct and meaningful summaries of the evidence on 
effectiveness of specific interventions.^^ 

The panel supported these summary statements with an impressive attention to the details and made recommendations 
about various areas where improvements are possible. In addition, the panel agreed with the view expressed by Board 
members and Hill staff that a broader review is needed in the future. 

As of September 2008, the WWC has released reports on 95 interventions that had evaluation studies that met WWC 
criteria with or without reservations ranging across the topics of beginning reading, character education, dropout 
prevention, early childhood education, English language learning, and elementary and middle school mathematics. 

(See table 13.) Of these interventions, 71 produced at least one positive or potentially positive outcome. Within each 
of these topic areas, the WWC has found interventions that have demonstrated positive or potentially positive effects 
on student outcomes. 



The members of the expert panel were Hendricks Brown, David Card (chair), Kay Dickersin, Joel Greenhouse, Jeffrey Kling, and Julia Littell. 
Expert Report on the What Works Clearinghouse. (2008) . pp. 10-11. The report is available at http://ies.ed.gov/director/board . 
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Table 13 



Identification of interventions that have positive or potentially positive effects by the What Works Clearinghouse 




Number 


Number of 


Number of interventions with 


Number of interventions 


Topic 


of studies 
reviewed 


interventions 
with studies 


studies meeting WWC criteria 
with or without reservations 


with at least one positive or 
potentially positive effect 


Beginning 

Reading 


987 


156 


24 


22 


Character 

Education 


93 


41 


13 


9 


Elementary 
School Math 


237 


73 


5 


1 


Middle School 
Math 


185 


35 


7 


5 


English 

Language 

Learners 


69 


32 


12 


11 


Dropout 

Prevention 


84 


22 


16 


12 


Early Childhood 
Education 


194 


46 


18 


11 


SOURCE: What Works Clearinghouse 


wehsite fhttp://ies.ed.p-ov/ ncee/wwc/ reports'). 





The Board recognizes, however, that for the evidence base to truly be useful to education decisionmakers and 
practitioners, it will need to cover a much broader range of topics and to cover each topic more in depth. For example, 
there is precious little information on which mathematics curricula are effective. According to the WWC’s elementary 
and middle school mathematics curricula reviews, the majority of curricula have yet to be evaluated with studies that 
meet WWC criteria. Many more research and evaluation projects are needed to fill the void. First, the research needs 
to be done. Second, the WWC needs to complete reviews of the research in many more topics. Currently, the WWC 
has reviewed interventions in seven areas; many other domains need to be addressed. To whom can a superintendent 
turn for information on science curricula, teacher professional development programs, social studies curricula, or 
high school mathematics curricula? Although the Board is pleased with the progress to date and the Board recognizes 
that creating the processes and procedures that are the working engine of the WWC took substantial thought, effort, 
and time, the Board encourages the Institute to press ahead to produce more content to make the WWC a one-stop 
resource center for education decisionmakers. 

C.l.b. What Works Clearinghouse/Practice Guides 

One of the early criticisms of the WWC was that very little research met WWC’s standards and, consequently, the 
WWC could not determine whether interventions worked. This situation left education leaders with little guidance 
for making decisions. Despite pressure to relax its standards and use results from weaker studies, the Institute 
maintained its standards for the WWC, but developed a new product — the practice guide. Practice guides provide 
practical recommendations for educators to help them address the everyday challenges they face in their classrooms. 



National Board for Education Sciences 5-Year 



Report, 2003 Through 2008 



26 





schools, and districts. Developed by a panel of nationally recognized experts, practice guides consist of actionable 
recommendations and strategies for overcoming potential roadblocks. Similar to practice guides in the medical 
field, the WWC practice guides also include the expert panel’s rating of the strength of evidence supporting each 
recommendation. Each practice guide goes through the external review process managed by the Institute’s Standards 
and Review Office. As of September 30, 2008, the WWC has released seven practice guides. As indicated in table 14 
below, these practice guides have been downloaded thousands of times from the WWC website. The practice guides 
provide the foundation for ED’s Doing What Works website. The Doing What Works website provides teachers with 
examples of possible ways of implementing the recommendations from the Institute’s practice guides. 

The practice guides reflect an important trait that the Institute has cultivated — adaptability. Over the past 6 years, 
the Institute has been responsive to external feedback in ways that maintain the rigor and quality of its research but 
recognizes the pragmatic importance of producing products that address the needs of the education decisionmakers 
and practitioners. The Institute has generated new products and improved the usability of existing products and 
services to better serve the education community. 

In the future, it will be important for the Institute to gather further evidence on the usefulness of these guides for 
practitioners. 



Table 14 


Practice Guides released by the What Works Clearinghouse 








Number of 


Number of 


Practice guide 


Date released 


recommendations 


downloads^ 


Effective Literacy and English Language Instruction 
for English Learners in the Elementary Grades 


July 2007 


5 


14,547 


Organizing Instruction and Study to Improve 
Student Learning 


September 2007 


7 


30,164 


Encouraging Girls in Math and Science 


September 2007 


5 


16,734 


Turning Around Chronically Low-Performing 
Schools 


May 2008 


4 


20,635 


Improving Adolescent Literacy: Effective Classroom 
and Intervention Practices 


August 2008 


5 


7,295 


Dropout Prevention 


September 2008 


6 


4,993 


Reducing Behavior Problems in the Elementary 
School Classroom 


September 2008 


5 


5,744 


' Downloads calculated from initial release of report through September 30, 2008. 
SOURCE: What Works Clearinghouse fhttp://ies.ed.gov/ncee/wwc). 



C.l.c. Interventions Supported by the Research Centers 

The Institute has established ambitious long-term goals for its research programs for producing Institute-supported 
interventions that are reported by the WWC to be effective in improving student outcomes by 2013-14. The targets 
are a minimum of 15 reading or writing interventions, 12 mathematics or science education interventions, and 10 
teacher-quality interventions that are shown to be effective when widely deployed. Annual targets have been set for the 
numbers of interventions with evidence of efficacy in improving student outcomes in small-scale, initial evaluations 
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using designs that meet the standards of the WWC.^"* The first targets were set for 2007, and the Institute met or 
exceeded those targets for identifying Institute-supported interventions that demonstrate positive findings in initial 
evaluations that meet WWC standards with or without reservations. 



Table 15 



Number of NCER-supported interventions producing positive effects when 
evaluated using designs that meet the What Works Clearinghouse criteria 

Read/ write Read/ write Math/science Math/science Teacher Teacher 

Year target actual target actual quality target quality actual 

2007 6 6 3 4 3 3 

SOURCE: U.S. Office of Management and Budget, Performance Assessment Rating Tool, Institute of Education Sciences Research report 
( http://www.whitehouse.gOv/omb/expectmore/summary/lQ009008.2007.html ). 



C.2. Is the Institute widely disseminating information? 

The Institute has created new systems (e.g., WWC) and revamped old systems (e.g., ERIC) for disseminating practical 
information to education leaders, practitioners, parents, and students. The Board applauds the progress the Institute 
has made in less than 6 years and highlights a few examples of the Institute’s efforts in this report. 

C.2.a. What Works Clearinghouse 

As described earlier, the creation of the WWC is a significant accomplishment. Nowhere else can education 
leaders find a transparent, unbiased report of the evidence on the effectiveness of education programs, practices, 
and policies. One key to its success has been its continued improvement since its inception. In the previously 
mentioned report commissioned by the Board to review the procedures of the WWC,^^ the panel identified areas 
for possible improvement. The Board believes that the WWC contractor and Institute staff should consider those 
recommendations and make changes accordingly. 

The WWC materials are also disseminated through other venues — in particular, on ED’s Doing What Works website. 
The Office of Planning, Evaluation, and Policy Development’s Doing What Works website is a practical resource that 
provides teachers with information on evidence-based practices. The Institute serves as a partner on this project by 
providing the Doing What Works team with WWC practice guides and other WWC reviews that identify practices that 
have sufficient scientific evidence to recommend to teachers. In short, the Institute provides the evidence base; the 
Doing What Works team uses videos and other visuals to provide actual examples of how to implement in classroom 
settings the recommendations. 

At this point, the Board believes that the WWC has developed sufficient products on its website for the WWC to 
develop a system to regularly collect data from its users on their perceptions of the usefulness of its reports. Do state 



Although initial evaluations are rigorously designed, they typically are not scale-up evaluations that provide evidence that the intervention can 
be widely deployed under typical conditions and produce an effect. Consequently, these evaluations do not necessarily meet the long-term goal 
of demonstrating effectiveness when widely deployed. 

Expert Report on the What Works Clearinghouse. (2008). The report is available at http://ies.ed.gov/director/board . 
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and local education decisionmakers find the intervention reports useful? Do they incorporate WWC information into 
their decisionmaking processes? The Board recommends that the Institute begin collecting such data. 

C.2.b. ERIC; Education Resources Information Center 

Another major accomplishment of the Institute is its improvement of ERIC, the world’s largest digital library of 
education research and information. When the contracts for the existing ERIC expired in December 2003, the 
Institute undertook redesigning ERIC to make it more efficient, more cost-effective, and fully capable of taking 
advantage of current Internet technologies. Today, ERIC provides free Internet-based access to bibliographic records 
of journal and nonjournal literature indexed from 1966 to the present. In 2003, ERIC materials such as microfiche 
files and electronic databases were sold to libraries, for-profit vendors, and individuals; today all ERIC materials 
and services are free. In 2003, a large portion of the ERIC library was available only on microfiche; today all of the 
holdings are digitized with the exception of about 340,000 older documents that cannot be converted due to the 
specific language of the permission forms under which access to the documents was originally granted.^'’ In 2003, 350 
journals were indexed in their entirety; in 2008, more than 650 journals are indexed comprehensively and another 150 
journals are indexed selectively.^^ In 2007, ERIC reached a new high in its customer user base with approximately 93 
million separate searches. In the first quarter of 2008 alone, 28,776,644 total searches were conducted through the 
ERIC database (up 4 percent from the previous quarter). 

C.2.C. College Navigator 

The Institute worked hard to improve its college search engine and released the new product on September 26, 2007, 
as the College Navigator website, which is designed to serve students and parents who are searching for information on 
colleges. College Navigator uses data collected by NCES through the Integrated Postsecondary Education Data System 
and the Office of Postsecondary Education. New features allow users to modify and fine-tune criteria without starting 
over, to build a list of favorites and conduct side-by-side comparisons, and to search by such specifics as distance 
from home, intercollegiate athletic programs, and size of school, to name a few. For adult learners, it finds programs 
that offer extended learning opportunities such as weekend or evening courses, distance learning, and credit for life 
experiences. College Navigator provides information on about 7,000 postsecondary institutions in the United States. 

Money Magazine named College Navigator as one of the magazine’s top 28 websites, one of the best places to start a 
college search, and the college search site with the best “first screen.” “Select basic constraints such as location, public 
or private college and major,” the Money article states, and College Navigator will “pump out a list of colleges. Click 
on any of them and you get a full set of the latest data on expenses, aid, enrollment, admissions and graduation rates, 
majors, and more, along with a Google map pinpointing location. There’s also a good comparison tool, and you 
can export data into an Excel sheet.”^® From its launch on September 26, 2007, through June 30, 2008, the College 
Navigator has received 700,302 visitors who made more than 5,891,901 page views. 



“ ERIC is contacting copyright holders for permission to disseminate the full text of these older materials online. 

See the ERIC website at http://ies.ed.gov/ncee/proiects/eric.asp . For journals that are selectively indexed, only education-related articles are 
included in ERIC. 

Light, J., Mangla, I.S., and Regnier, E (2007, December 4). Click Your Way to the Right School. Money Magazine. Retrieved July 16, 2008, 
from http://money.cnn.eom/2007/l 1 /29/pf/bestweb college. moneymag/index.htm . 
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C.2.d. Other Activities 



The Institute supports a variety of other activities to disseminate findings of rigorous education research. As 
discussed in section A.l.c., the RELs are a major vehicle for dissemination of research findings. Another example of 
dissemination work is the American Institutes for Research’s Scientific Evidence in Education Forum (SEE Forum), 
which is funded by the Institute. The SEE Forum presents key findings from timely, rigorous research, enabling 
policymakers and education stakeholders to make well-informed decisions, and works to build greater awareness 
regarding the value of scientific evidence in education. To date, three forums have been held — one on turning 
around low-performing schools, one on effective literacy and English language instruction, and one on the impact of 
afterschool interventions. The Forums are held in Washington, DC, to provide a critical information link between 
rigorous research and the Washington, DC-based policymaking community. A website at http://www.seeforums. 
org has been developed to provide broad-scale access to the information presented in the forums and support 
policymakers’ efforts to apply rigorous evidence. 

C.3. Is the Institute helping education ieaders and decisionmakers obtain better data 
on their education programs and education achievement in their states and 
districts? Is the Institute heiping education ieaders and decisionmakers become 
better consumers of education research? 

C.3.a. Statewide Longitudinal Data System (SLDS) 

One of the Institute’s most important programs for improving the data that states and their districts have is the SLDS 
grant program, which is administered by NCES. Through this program, grants are provided to support the work of 
State Education Agencies (SEA) as they design, develop, and implement statewide longitudinal data systems that will 
assist states in improving student achievement and reducing achievement gaps between different groups of students 
in kindergarten through grade 12. SLDS grants support long-term endeavors to increase the number, capacity, and 
functionality of these systems. The grants promote interoperability among school districts within a state, between 
states, and between elementary and secondary education systems and postsecondary education. 

Twenty-seven states received funding totaling $1 14,848,893 for FY 06 and FY 07. Approximately $94 million in 
funding has been appropriated for FY 09. 
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Table 16 



Statewide Longitudinal Data Systems awards 


FY 06 grantees 


FY 07 grantees 


Alaska 


Arizona 


Arkansas 


Colorado 


California 


District of Columbia 


Connecticut 


Indiana 


Florida 


Kansas 


Kentucky 


Maine 


Maryland 


Nebraska 


Michigan 


Nevada 


Minnesota 


New Hampshire 


Ohio 


North Carolina 


Pennsylvania 


Oregon 


South Carolina 


Utah 


Tennessee 


Virginia 


Wisconsin 


SOURCE: lES National Center for Education Statistics, Statewide 
Longitudinal Data Systems grant program (http://nces.ed.gov/ 
proframs/slds/1. 



In addition to administrative grants management activities, NCES provides technical assistance to grantees, and 
promotes the exchange of ideas among grantees through a wide range of activities. Grantees have opportunities to 
learn from experts and from each other through opportunities such as the annual 2-day conference hosting at least 
two members from each grantee’s project team, monthly conference calls with grantee project directors and monthly 
“webinar” discussions hosted by a grantee team for all grantee recipients. Technical assistance is provided to grantee 
states through a contract with the Council of Chief State School Officers (e.g., technical assistance provided in 
adopting XML data standards). NCES maintains a program website that includes helpful links and PowerPoint files 
of all past presentations and a document depository resource for sharing and distributing SLDS-related products and 
materials developed by the grantee states. In addition, a Longitudinal Data Systems Handbook is being developed 
through NCES’s Forum on Education Statistics to synthesize all major components of developing, implementing, 
and using such systems. The Board strongly commends the Institute for developing the program and providing the 
necessary support to enable SEAs to produce longitudinal data systems that will support a sophisticated education 
epidemiology that will be the engine for hypothesis generation and refinement for what works and what does not in 
state and local education policy. 
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C.3.b. National Forum on Education Statistics 



The purpose of the Institute’s National Forum on Education Statistics (Forum) is to improve the quality, 
comparability, and usefulness of data used by all levels of government for decisions about elementary and secondary 
education. The members include a representative of each SEA, and one local education agency in each state, appointed 
by the chief state school officer, and representatives of ED program offices and other federal agencies and professional 
associations that have an interest in education data (approximately 125 members). The Forum serves as a sounding 
board for ED policies about elementary and secondary data (e.g., developing a realistic timeline for implementing the 
Office of Management and Budget’s 1997 race and ethnicity standards among states and school districts). It comprises 
a community of education data system managers, encouraging innovation and providing technical assistance to the 
members. The Forum’s most important function has been to identify areas of weakness or ambiguity in education data 
management and use, and to create “best practice” guides to resolving these issues. The Forum Guides are written for 
data system professionals, state and local administrators, and in some cases, teachers and parents. Examples of this 
work include Forum Guides on protecting the confidentiality of student data; collecting information about virtual 
education; classifying student enrollment status in a standard taxonomy; and a recent curriculum guide to improving 
data quality, directed toward school and district personnel. The Board is pleased with the Institute’s leadership in 
facilitating the development of education data systems that can be utilized to provide useful and timely answers for 
education leaders and decisionmakers. 

C.3.C. NCES Summer Data Conference 

NCES sponsors this 3-day conference every July in the Washington, DC, area. There are usually more than 500 
attendees. The purpose of the Data Conference is to update federal, state, and local data professionals on issues and 
innovations in data collection, management, and use. The NCES Summer Data Conference differs from other ED 
conferences in that it is directed at education management information systems as a whole, rather than data related 
to a single federal program. The July 2008 Data Conference included more than 90 concurrent sessions, a number 
of them concerned with building and using longitudinal student data systems, implementing the ED’s EDFacts data 
collection system, or adopting electronic data standards. There were presentations by state and school district staff 
on topics such as using data to improve instruction, improving data quality, building a data dictionary, and using 
longitudinal data to evaluate the effectiveness of teacher preparation programs. 

C.3.d. International Assessments 

Making international assessment results understandable and useful to the press and the public has been an ongoing 
feature of recent NCES releases. For example, in late 2007, prior to the releases of the latest round of results from the 
Progress in International Reading Literacy and from the Program for International Student Assessment (PISA), NCES 
conducted prebriefings with the press that explained how international assessments compare in purpose, samples, 
and content with NAEP and to each other, so that differences in performance and trends across assessments could be 
interpreted knowledgeably. NCES has also recently convened expert panels to examine the content of international 
assessments against state standards through the work of the Council of Chief State School Officers and in relation to 
the recommendations of the National Mathematics Advisory Panel. These efforts are intended to extend the usefulness 
of international assessments beyond the horse race of relative standings, to begin to pose questions of U.S. curriculum 
rigor in comparison to that of high-performing countries. In the area of adult literacy, NCES has led efforts to 
establish a new international assessment that borrows from and is timed to alternate with its own national assessment 
in the United States, so that measures of adult literacy are available on a regular 5-year cycle. 
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In 2008, NCES conducted a meeting for state and national education leaders, including, among others, chief state 
school officers, state legislators, representatives from the National Governors Association, and congressional staff, 
to provide information about the Trends in International Mathematics and Science Study (TIMSS) and PISA and 
discuss how each assessment relates to state standards and assessment activities, the analytic possibilities and limitations 
of the international assessments, and other options for obtaining international benchmarks. In the past, U.S. states 
have been invited to participate as individual entities in TIMSS; 15 have done so at some point, and as many as 13 
have participated in any one administration (1999). In the future, states will be invited to participate in both TIMSS 
(in 2011) and PISA (in 2009 and 2012). The Institute, through NCES, has taken a leadership role in helping states 
understand the costs, benefits, and alternatives to participating in international assessments for states wanting to 
determine how their students compare to students in other countries. 

C.3.e. Other Activities 

The Institute has sponsored a number of programs to help education leaders and decisionmakers understand education 
research or become more discerning consumers of education research. For example, the Institute has recognized that 
evidence-based answers to support all of the decisions that education decisionmakers and practitioners must make 
every day do not yet exist. Moreover, education leaders cannot always wait for scientists to provide answers. One 
solution to this dilemma is for the education system to integrate rigorous research and evaluation into the core of its 
activities. In spring 2008, the Institute sponsored a workshop on evaluation of state and local education programs and 
policies, with 135 people participating in the program. For this workshop, the focus was on providing staff from state 
and local education agencies with a general understanding of how a rigorous impact evaluation could be embedded in 
the implementation of new programs or existing programs that are not yet universal in implementation. 

Another example is the Institute’s collaboration with the National Conference of State Eegislatures in conducting a 
series of seminars to inform state legislators and their staff about research on specific education issues. At these training 
institutes, research is translated into practical recommendations on topics starting with teacher effectiveness and 
dropout prevention. Three seminars in 2007 focused on linking policy and research on recruiting and retaining high- 
quality teachers. Two seminars for legislators attracted 37 legislators from 30 states; one seminar for legislators attracted 
40 participants representing 22 states. Three seminars in 2008 focus on preventing and supporting school dropouts. 
Two seminars for legislators attracted 47 legislators from 37 states; a forthcoming seminar for legislative staff is 
expected to reach 30-35 attendees. In addition to state legislators and their staff learning about the findings of current 
education research, these seminars have provided Institute leaders with opportunities to hear about the questions for 
which state leaders would like answers from education scientists. 

The Board commends the Institute for seeking opportunities to engage education leaders and decisionmakers on 
current issues and providing them with unbiased presentations based on scientific research. The Board strongly 
encourages the Institute to develop programs to enable education leaders and practitioners to better understand and 
integrate scientifically based evidence into their decisionmaking. There is a lack of understanding in the broader 
education community of what constitutes solid research. Analogous to the need to train researchers to conduct 
rigorous education research, there is a need to train education decisionmakers and practitioners to recognize what 
constitutes high-quality research and evidence and understand why making distinctions in the quality of research is 
important. The vast majority of education leaders and practitioners received their knowledge of research from those 
whose research skills are relatively limited by todays standards. The Institute must take an active role in educating 
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the education community so that it understands that all research is not the same. Some research provides valid 
information about causal relations; other research may be just interesting correlational data. If the Institute does not 
help the education community understand this distinction, the education community will continue to be vulnerable 
to false claims of causal evidence. 
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Overall Conclusions and 
Recommendations From the National 
Board for Education Sciences 



When the Institute was established in November 2002, many in the education research and policy community cried 
deja vu. As some observed, why would anyone expect the Institute to accomplish what its predecessors — the National 
Institute of Education and OERI — had not? The members of the Board conclude that in a relatively brief period of 
time the Institute has made exceptional progress in improving the rigor and relevance of education research in our 
nation. Under the leadership of its first director, Grover J. Whitehurst, the Institute has accomplished what many 
believed could not be done. 

The framework for this nonideological, high-quality work was wisely established by Congress in the ESRA. Because 
that Act has generated such strong results, the Board is recommending its rapid reauthorization, with a set of modest 
amendments intended to clarify and strengthen it. In addition, the Board recognizes that transformation of education 
into an evidence-based field is an enormous task. It will need to involve everyone from federal and state policymakers 
to local education leaders, administrators, teachers, and parents. By analogy, the progress to date is akin to changing 
the course of an enormous ocean liner. Over the past 6 years, directional change has been accomplished. We now need 
to stay on course to get to the destination. 

In the next section, the Board summarizes its conclusions and recommendations for the Institute going forward. Above 
all, the Board asserts that it is most important that the Institute stay the course — that the standards, procedures, and 
policies that have yielded this new generation of rigorous and relevant research and evaluation continue as the Institute 
moves from its genesis to firmly establishing itself as a science agency on par with NIH and NSE 

A. Summary of Conclusions and Recommendations 
A. 1 . Advancing the Rigor of Education Research 

In its first 6 years, the Institute has made its greatest advances in improving the rigor of education research and 
evaluation. Although the quality of research is not sufficiently improved to thoroughly transform education into an 
evidence-based field, this transformation cannot be accomplished without this necessary first step. It is the Board’s 
opinion that it is absolutely crucial that going forward, the Institute maintain the high standards it has established for 
education research. The only way for research to improve student outcomes is for the research to be of high quality. 

The Board is proud of the Institute’s scientific review system. This system and the standards it maintains are the 
foundation for improving the rigor of education research, evaluation, and statistics. 

Based on a review of ratings of the quality of education research projects by an independent panel of eminent 
scientists, the Board commends the Institute for improving the scientific quality of education research. The Board 
is pleased that over its first 6 years the Institute has achieved a substantial increase in the number of research grants 
funded each year and the number of applications received. 
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Given concerns that have been raised with regard to the Institute narrowly focusing its research grant funding on 
projects that utilize RCTs to evaluate the impact of interventions, the Board was surprised to discover the diversity 
of the Institute’s research grant portfolio. About 26 percent of the grants funded under the education research 
grant programs and 23 percent of the grants funded under the special education research programs are evaluations 
of the impact of interventions for which the Institute prefers experimental designs. The Institute solicits research 
that addresses a broad range of questions, from exploring the relations between malleable factors (e.g., student 
competencies, education practices) and education outcomes to testing the validity and reliability of assessments, 
and encourages the use of a variety of methodologies — descriptive, observational, interview, qualitative, quasi- 
experimental, and experimental. The Board highly commends the Institute for addressing a wide range of education 
questions that are relevant to policy and practice and for ensuring that the questions are addressed through appropriate 
research methods. 

The Board applauds the Institute for developing a first-class evaluation program. Not only has the Institute 
implemented a number of rigorous evaluations of important education programs, but it also has initiated a process for 
using the findings of an evaluation to modify existing programs to improve outcomes for students. 

The Board is pleased with the Institute’s progress in establishing processes to improve the quality of the research 
conducted through the REL program. As the RELs move toward completing their 5-year contracts, the Board 
recommends that the Institute conduct an evaluation of the RELs including feedback from state and local education 
policymakers and decisionmakers on REL activities and reports. 

Based on its review of the Institute’s predoctoral and postdoctoral research training programs and its research and 
statistics training workshops, the Board concludes that the Institute has been successful in producing new education 
scientists with the necessary skills to conduct rigorous education research and in enhancing the skills of current 
researchers. The Board congratulates the Institute on its efforts and recommends that the Institute continue to invest 
in its research training programs. 

A.2. Increasing the Relevance and Usefulness of Education Research 

No matter how technically sound research activities might be, if they do not address the issues and questions that are 
of concern to education policymakers and practitioners, the research will not be used to inform education policy and 
practice. The Institute has made progress in improving the relevance and usefulness of education research, evaluation, 
and statistics. 

Based on ratings of the relevance of funded research projects by education leaders and administrators, the Board is 
generally pleased with the progress the Institute has made in improving the relevance of its education and special 
education research portfolios. The Institute accomplished a major improvement in the structure of federal education 
research programs by establishing focused, long-term programs of research on fundamental education outcomes (e.g., 
reading, writing, mathematics, and science achievement) and conditions (e.g., quality of the education workforce, 
education systems and policies). The Institute’s long-term programs of research are sufficiently broad (e.g., improving 
reading curricula, instructional practices, and assessments from prekindergarten through adult education) to cover 
a wide range of needs, while at the same time are targeted to expedite the accumulation of knowledge at a faster 
pace than the diffuse research portfolios typically generated from field-initiated research competitions. The Board 
recommends that the Institute continue to support its focused programs of research. The Board also recommends that 
the Institute continue to seek input from education decisionmakers on the issues for which they need answers. 
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The Board is pleased with the direction and practical focus of the research and dissemination work of the national 
research and development centers. The Institute has used the centers to generate relevant evidence in key policy areas 
(e.g., teacher incentives and disincentives, teacher credentials, assessment of English language learners). 

The Board applauds the Institute for dramatically improving the timeliness of the release of NCES data and is pleased 
that perceptions of the relevance of NCES reports continues to remain high. 

In reviewing the data on the timeliness of the release of NCEE evaluation reports, the Board recognizes that the 
Institute has been working to reduce the time from the preparation of the initial draft report to release of the approved 
document and has made some progress to that end. In addition to the strategies the Institute has already implemented, 
the Board recommends that the Standards and Review Office create a document that details the Institute’s standards 
and provides guidance to those preparing Institute reports. In addition, the Board recommends that NCEE establish 
procedures to ensure timely receipt of draft reports from its contractors and to ensure efficient handling of drafts by 
NCEE staff. 

Although the Institute has some promising data on the relevance of NCEE projects, it has not yet established 
procedures for collecting data on the relevance of NCEE reports. The Board recommends that the Institute begin 
collecting data on the relevance of NCEE evaluation reports. 

The Institute has actively sought opportunities to engage education leaders and decisionmakers on current issues and 
provide them with unbiased presentations based on scientific research. The Board commends the Institute for its 
work and encourages the Institute to continue to seek ways to enable education leaders and practitioners to integrate 
scientifically based evidence into their decisionmaking. 

A.3. Facilitating Evidence-Based Decisionmaking 

Generating rigorous and relevant research is a necessary step toward transforming education into an evidence-based 
field. However, the Institute must then disseminate the knowledge it produces to facilitate its successful application 
by education policymakers and practitioners. The Institute has made progress in this arena, but should continue to 
develop venues and products to support this effort. In addition, the Institute needs to collect data on stakeholders’ 
perceptions of the usefulness of its products and services. 

Based on examination of WWC intervention reports and practice guides, data provided on interventions that have 
been supported through research grants, and NCEE evaluation reports, the Board is impressed with the progress the 
Institute has made over the past 6 years in starting a new generation of research findings that can inform education 
decisionmaking. Because the Institute has not collected much data on the degree to which education policymakers 
and decisionmakers actually use the information, the Board recommends that the Institute begin collecting data 
on stakeholders’ perceptions of the usefulness of WWC products and NCEE evaluation reports for informing their 
decisions. 

Based on information about the WWC, College Navigator, and other programs, the Board believes that the Institute 
has created dissemination mechanisms with an eye toward serving education leaders, practitioners, parents, and 
students. The Board commends the Institute for its accomplishments. The Board recommends that the Institute 



® Existing relevance data are ratings of the relevance of the project plans and were obtained prior to the execution and completion of the 
evaluations. 
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regularly collect data regarding users’ perceptions of the usefulness and usability of the information available through 
these programs and how they are using the information in their decisionmaking. 

Based on information on the Institute’s SLDS program, its Forum, the NCES Summer Data Conference, and its work 
on international assessments, the Board finds that the Institute provides a wide range of technical assistance relative to 
helping states and districts develop, understand, and use administrative and assessment data. The Institute’s statistics 
center, with a long history upon which to draw, can serve as a model for the other Centers for the type of specialized 
technical assistance that is unlikely to be available through other venues. That said, the Board would like the Institute 
to regularly collect data on users’ perceptions of the usefulness of these services. 

The Board commends the Institute for seeking opportunities to engage education leaders and decisionmakers on 
current issues and providing them with unbiased presentations based on scientific research. The Board strongly 
encourages the Institute to develop programs to enable education leaders and practitioners to better understand what 
constitutes solid research and the reasons why making distinctions in the quality of research is important. The vast 
majority of education leaders and practitioners received their knowledge of research from those whose research skills 
are relatively limited by today’s standards. The Institute must take an active role in educating the education community 
so that it understands that all research is not the same. Some research provides valid information about causal relations; 
other research may be just interesting correlational data. If the Institute does not help the education community 
understand this distinction, the education community will continue to be vulnerable to false claims of causal evidence. 

B. Key Recommendations Identified in Mark-up of Education Sciences Reform Act 

At its January and May 2008 meetings, the Board discussed suggestions for revising ESRA. The Board believes that 
ESRA has been instrumental in creating the framework that has encouraged nonideological, rigorous, and relevant 
educational research. The Act should be reauthorized quickly, with modest changes that will make it clearer and 
stronger. Appendk F is an annotated version of ESRA that includes the Board’s suggestions and rationale for the 
proposed change. The types of changes include (a) clarifying ambiguous descriptions; (b) establishing processes to 
avoid long-time vacancies; (c) maintaining the peer review process under the office of the Director; (d) establishing 
procedures to buffer the Institute further from outside influences to maintain its integrity and independence as a 
science agency; (e) changing the definition of scientifically based research to align with more recent language in other 
legislation; and (f) establishing the Institute as the lead agency for congressionally authorized scientific education 
research under various education laws. 

C. Resolutions of the National Board for Education Sciences 

Since inception, the Board has approved the following resolutions. They are repeated here as guidance for the 
Congress, the Secretary, and the Director; 

1 . Congress should designate the Institute, in statute, as the lead agency for all congressionally authorized 
evaluations of ED programs, responsible for all operations, contracts, and reports associated with such 
evaluations. (September 2006) 

2. Congress should allow ED to pool funds generated by the 0.5 percent evaluation set-aside from smaller 
programs. (September 2006) 
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3. ED should use its “waiver” authority to build scientifically valid knowledge about what works in K-12 
education. (September 2006) 

4. Congress, in authorizing and funding evaluations of federal education programs, should require [program] 
grantees, as a condition of grant award, to participate in the evaluation if asked, including the random 
assignment to intervention and control groups as appropriate. (April 2005) 

5. Congress should create, in statute, effective incentives for federal education program grantees to adopt 
practices or strategies meeting the highest standard of evidence of sizeable, sustained effects on important 
educational outcomes. (May 2007) 

6. Congress and ED should ensure that individual student data can be used by researchers (with appropriate 
safeguards for confidentiality) in order to provide evaluations and analyses to improve our schools. (September 
2006) 

7. Congress should revise the statutory definition of “scientifically based research” so that it includes studies 
likely to produce valid conclusions about a program’s effectiveness, and excludes studies that often produce 
erroneous conclusions. (October 2007) 

8. The Board reviews and advises the Institute director on grant awards where the proposed grantee is selected 
out of rank order of applicant scores that result from peer review for scientific merit. (January 2008) 

9. The Board commends the Secretary and ED for moving forward in developing new regulations and guidance 
about how to maintain confidentiality of educational data under Family Educational Rights and Privacy 

Act (FERPA) while also providing for research uses of student and school data. ED should finalize these 
regulations quickly, incorporating the major clarifications that have been submitted in comments. (May 2008) 

10. Congress should expand on the program of supporting statewide longitudinal data systems by requiring 
that states accepting funding under this program agree to make data in these systems available to qualified 
researchers (subject to FERPA) for the purpose of research that is intended to help improve student 
achievement. (May 2008) 
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Appendix A 

Types of Projects Funded by the Institute of Education Sciences 

Goal 1 : Exploration and Identification Projects 

The purpose of Exploration and Identification projects is to explore the relationship between malleable factors 
(i.e., factors that can be changed, such as student characteristics and education practices) and education outcomes 
to identify potential targets of intervention; and explore factors and conditions that may mediate or moderate the 
relations between malleable factors and education outcomes. For these projects, the Institute encourages a number 
of methodological approaches including secondary data analyses, observational studies, and meta- analyses of existing 
studies. 

Goal 2: Development Projects 

Under the Development goal, applicants propose to develop new education interventions (e.g., instructional 
approaches, curricula, professional development programs for teachers or education leaders). Applicants may propose 
to use a variety of methodological approaches to develop the intervention, assess the feasibility of implementing the 
newly developed intervention in an authentic education delivery setting, and show the promise of the intervention 
for achieving the desired outcome (e.g., improving math learning). In the most recent Request for Applications, the 
Institute notes that observational, survey, or qualitative methodologies are encouraged to examine the implementation 
of the intervention. 

Goal 3: Efficacy and Replication Projects 

The Institute funds Efficacy and Replication projects to determine whether or not fully developed interventions — 
programs, practices, and policies — are effective under specified conditions (e.g., urban schools with a high turnover 
rate among teachers), and with specific types of students (e.g., English language learners). Based on the Request 
for Applications, the Institute expects Efficacy and Replication projects to use multiple methods for determining 
whether an intervention produces its intended outcomes and identifying the processes through which the 
intervention acts, as well as factors that facilitate or hinder the intervention. Applicants are encouraged to utilize 
random assignment designs to determine the impact of the proposed intervention on its intended outcomes and to 
incorporate observational, survey, or qualitative methods to help identify factors that may explain the effectiveness or 
ineffectiveness of the intervention. 

Goal 4: Scale-up Evaluations 

Under the Scale-up goal, the Institute funds projects to determine the impact of interventions that are implemented 
at scale and under real-world conditions (i.e., when they are implemented under conditions that would be typical if 
a school district or other education delivery setting were to implement them [i.e., without special support from the 
developer or the research team]). The methodological requirements are similar to those for the Efficacy and Replication 
goal. 
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Goal 5: Measurement Projects 

The Institute solicits projects to develop and validate instruments to measure student outcomes or to assess knowledge 
and practice of education professionals, as well as proposals to develop or test new techniques for assessment or 
analysis of assessment data in the context of state accountability standards and systems or to develop and validate cost- 
accounting tools. In its funding announcement, the Institute does not specify particular methodologies other than to 
indicate that applicants must describe the proposed methods for developing the measurement instrument, assessing 
the reliability and validity of the instrument, and analyzing data. 
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Appendix B 



National Center for Education Evaluation and 
Regional Assistance Impact Evaluations 



Table B-1 


List ofNCEE Impact Evaluations 




Title of Evaluation 


Design 


An Evaluation of the Impact of Mandatory Random Student Drug Testing 


RCT** 


An Evaluation of the Impact of Supplemental Literacy Interventions in Freshman Academies* 


RCT 


Closing the Reading Gap* 


RCT 


Effects of Transitional Bilingual Education, Two-Way Bilingual, and Structured English Immersion 
Programs and the Literacy and Oracy of Spanish-Dominant Children 


RCT 


Evaluation of Conversion Magnet Schools 


ITS*** 


Evaluation of the Effectiveness of Educational Technology Interventions* 


RCT 


Evaluation of the Impact of Charter School Strategies 


RCT 


Evaluation of the Impact of Literacy Instruction on Adult ESL (English as a Second Language) Learners 


RCT 


Evaluation of the Impact of the DC Choice Program* 


RCT 


Evaluation of Math Curricula 


RCT 


Evaluation of Reading Comprehension Programs 


RCT 


Even Start Classroom Literacy Interventions and Outcomes (CLIO) Study* 


RCT 


Feasibility and Conduct of an Impact Evaluation of Title I Supplemental Educational Services 




Feasibility and Impact Evaluation of Moving High-Performing Teachers to Low-Performing Schools 


RCT 


Impact Evaluation of a School-Based Violence Prevention Program 


RCT 


Impact Evaluation of Academic Instruction for Afterschool Programs* 


RCT 


Impact Evaluation of Response to Intervention Strategies 


RCT 


The Evaluation of Teachers Trained Through Different Routes to Certification 


RCT 


Impact Evaluation of Teacher Induction Programs* 


RCT 


Impact Evaluation of the U.S. Department of Education’s Student Mentoring Program 


RCT 


Impacts of School Improvement Status on Students With Disabilities 
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Table B-1 (continued) 



List ofNCEE Impact Evaluations 

Title of Evaluation 


Design 


Optimizing Educational Outcomes for English Eanguage Learners 


RCT 


Project ELLA (English Language/Literacy Acquisition) 


RCT 


Reading First Impact Study* 




The Impact of Professional Development Models and Strategies on Teacher Practice and 
Student Achievement in Early Reading* 


RCT 


The Impact of Professional Development Strategies on Teacher Practice and Student 
Achievement in Math 


RCT 


*Interim or final report is available for this evaluation on the NCEE website at http://ies.ed.pov/ncee/pubs/. 
**Randomized controlled trial. 

***lnterrupted time series. 

****Regression discontinuity design. 

SOURCE: National Center for Education Evaluation and Regional Assistance. 
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Appendix C 

List of Regional Educational Laboratories Randomized Controlled Trials Projects 

Under Current Contracts, 2006-11 

1 . The Effectiveness of Hybrid Secondary School Courses on Teacher Practices, Classroom Quality and Adolescent 
(RET Appalachia) 

2. An Impact Evaluation of Early Literacy Programs (REE Appalachia) 

3. Study of the Use of Classroom Assessment for Student Learning (REE Central) 

4. A Study of the Differential Effects of ELL Training and Materials (REE Central) 

5. A Study of the Effectiveness of a School Improvement Intervention (Success in Sight) (REE Central) 

6. The Effect of Connected Mathematics Program 2 (CMP2) on the Math Achievement of Middle School Students 
in Selected Schools in the Mid-Atlantic Region (REE Mid-Atlantic) 

7. The Effects of Compass Learning’s Odyssey Math Software on the Mathematics Achievement of Fourth-Grade 
Students (REE Mid-Atlantic) 

8. Improving Adolescent Literacy Across the Curriculum; An Evaluation of the Strategic Instruction Model’s 
Content (REE Midwest) 

9. Efficacy of Frequent Formative Assessment for Improving Instructional Practice and Student Performance (REE 
Midwest) 

10. Eighth-Grade Access to Algebra I: A Study of Virtual Algebra (REE Northeast) 

11. Accelerating Literacy in the Middle School (Thinking Reader) (REE Northeast) 

12. An Investigation of the Impact of a Traits-Based Writing Model on Student Achievement (REE Northwest) 

13. Study of Effectiveness of Project CRISS on Reading Comprehension and Achievement for High School Students 
in Grades 9 Through 10 (REE Northwest) 

14. Evaluation of Principles-Based Professional Development to Improve Comprehension for ELLs (REE Pacific) 

15. The Effectiveness of a Program to Accelerate Vocabulary Development in Kindergarten (REE Southeast) 

16. The Effectiveness of the Alabama Math, Science, and Technology Initiative (AMSTI) (REE Southeast) 

17. Improving the Comprehension and Vocabulary Skills of ELLs in Fifth Grade Using Collaborative Strategic 
Reading (REE Southwest) 

18. Randomized Evaluation of Tier 2 Mathematics Interventions for Multitiered Early Intervention Models in Grades 
(REE Southwest) 

19. Utilizing Lexiles for a Summer Reading Program for Low-Income Students (REE Southwest) 

20. Quality Teaching for English Language Learners (REE West) 
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21. Understanding Science and the Academic Literacy of English Language Learners (REL West) 

22. HS Instruction in Problem-Based Economics (REL West) 

23. Program for Infant/Toddler Caregivers (REL West) 

24. Lessons in Character Education (REL West) 

25. Assessment Accommodations for English Language Learners (REL West) 
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Appendix D 



Institutions Receiving Awards to Establish Predoctoral and 
Postdoctoral Research Training Programs 



Table D-1 




FY 04 and FY 05 Predoctoral Research Training Programs* 


Fiscal year (FY) 


University 


Training program director 


2004 


Carnegie Mellon University 


David Klahr 


2004 


Florida State University 


Christopher Lonigan 


2004 


Northwestern University 


David Uttal 


2004 


University of Virginia 


Robert Pianta 


2004 


Vanderbilt University 


David Cordray 


2005 


University of Chicago 


Stephen Raudenbush 


2005 


University of Miami 


Daryl Greenfield 


2005 


University of Minnesota 


Mark Davison 


2005 


University of Pennsylvania 


Rebecca Maynard 


2005 


University of Wisconsin 


Adam Gamoran 


*In FY 08, the Institute held another competition for predoctoral research training programs. Three new institutions received awards (Johns 
Hopkins University, New York University, and University of California, Los Angeles) and two of the FY 04 grantees received a second 5-year 
predoctoral award (University of Chicago, Vanderbilt University). The data included in this report do not include information from the FY 08 
awards because these predoctoral programs begin in the summer of 2008. 

SOURCE: lES/NCER Training Programs in Education Sciences 2004-2008 Report. 



National Board for Education Sciences 5-Year 



Report, 2003 Through 2008 



D-1 








Table D-2 




Fiscal year (FY) 


FY 05— FY 07 Postdoctoral Training Programs* 

University 


Training program director 


2005 


Florida State University 


Christopher Lonigan 


2005 


Northwestern University 


Larry Hedges 


2005 


Purdue University 


Douglas Powell 


2005 


University of California, Berkeley 


Prentice Starkey 


2005 


University of Pittsburgh 


Charles Perfetti 


2005 


Vanderbilt University 


David Cordray 


2006 


Stanford University 


Tony Bryk 


2006 


University of North Carolina 


Donna Bryant 


2006 


University of Oregon 


Rob Horner 


2006 


University of Virginia 


Robert Pianta 


2007 


Georgia State University 


Rose Sevcik 


2007 


University at Buffalo, SUNY 


Julie Sarama 


2007 


University of California, Berkeley 


Mark Wilson 



*In FY 08, the Institute held another competition for postdoctoral research training programs. Four principal investigators received awards 
(Bruce Homer and Jan Plass, City University of New York/New York University; Susan Sheridan, University of Nebraska-Lincoln; Gerald 
Tindal, University of Oregon; and Paul Cobb, Vanderbilt University). The data included in this report do not include information from the FY 
08 awards because these postdoctoral grants begin in the summer of 2008. 

SOURCE: lES/NCER Training Programs in Education Sciences 2004-2008 Report. 
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Appendix E 

Predoctoral and Postdoctoral Research Training Programs 



During their training programs, the predoctoral and postdoctoral fellows are beginning their careers as productive 
researchers. In the last 2 years, predoctoral fellows have averaged two research presentations at peer-reviewed 
conferences per year and had four publications per year. For the same time period, postdoctoral fellows have averaged 
three research presentations at peer-reviewed conferences per year and one publication per year. 



Table E-1 



Number of conference presentations and publications by Institute fellows 





Number 


Number 


Number 


Number 


Total number 


Total number 




refereed 


refereed 


papers 


papers 


refereed 


papers 




conference 


conference 


published/ 


published/ 


conference 


published/ 




presentations, 


presentations, 


in press 


in press 


presentations 


in press 


Institute 


7/1/06- 


7/1/07- 


7/1/06- 


7/1/07- 


7/1/06- 


7/1/06- 


fellow 


6/30/07 


3/1/08 


6/30/07 


3/1/08 


3/1/08 


3/1/08 


Predoctoral* 


307 


355 


57 


69 


662 


126 


Postdoctoral** 


53 


79 


16 


36 


132 


52 



*143 active fellows 2006-07, 189 active fellows 2007-08. 

**18 active fellows 2006-07, 26 active fellows 2007-08. 

SOURCE: lES/NCER Training Programs in Education Sciences 2004-2008 Report. 
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Appendix F 

Annotated Version of the Education Sciences Reform Act With 
Recommendations for Changes From the National Board for Education Sciences 
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EDUCATION SCIENCES REFORM 



PUBLIC LAW 107-279 

AS AMENDED BY PUBLIC LAW 108-446, THE 
INDIVIDUALS WITH DISABILITIES EDUCATION 
IMPROVEMENT ACT OF 2004, ENACTED ON 
DECEMBER 3, 2004; 118 STAT. 2647 



Notes 



1 . P.L. 107-279 is often referred to as the "Education Sciences Reform Act of 2002" 
(ESRA). That short title, however, applies only to Title I of the statute. There are 
separate short titles for Title II (the "Educational and Technical Assistance Act of 
2002") and Title III (the "National Assessment of Educational Progress Authorization 
Act"). There is no overall short title for the entire statute. 

2. The amendments highlighted in this document were made by section 201 (a)(1 ), 
(a)(2), (b)(1), and (b)(2); section 202; and section 203 of P.L. 108-446, the 
Individuals with Disabilities Education Improvement Act of 2004; see 118 Stat. 

2799 - 2803. Section 302(c)(1 ) of that Act provides that new sections 175, 176, and 
177 (except for subsection (c)) of the ESRA took effect on enactment (Dec. 3, 

2004). Section 302(c)(2) provides that section 177(c) of the ESRA takes effect on 
October 1, 2005; see 118 Stat. 2803. In addition, section 303 of P.L. 108-446 
provides various authorities and instructions to the Secretary of Education to ensure 
the orderly transition of the amendments made by that Act, including the 
amendments to the ESRA; see 1 1 8 Stat. 2803 - 2804. 

3. This document sets out Titles I, II, and III of P.L. 107-279, as amended. It does not 
include Title IV of the Act, which was itself amendatory and has been executed. 



45. This document was prepared by the Division of Legislative Counsel in the Office of 
the General Counsel, U.S. Department of Education as a service for ED employees. 
It is not an official statement of the law and should not be relied on for that purpose. 
The official version of the law is set out in the Statutes at Large and in the U.S. 
Code. 
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Public Law 107-279, Novembers, 2002; 116 Stat. 1940 

An Act 

To provide for improvement of Federal education research, statistics, evaluation, 
information, and dissemination, and for other purposes. 

Be it enacted by the Senate and House of Representatives of the United States of 
America in Congress assembied, 

SECTION 1. TABLE OF CONTENTS. 

The table of contents for this Act is as follows: 

Sec. 1. Table of contents. 

TITLE I - EDUCATION SCIENCES REFORM 

Sec. 101. Short title. 

Sec. 102. Definitions. 

Part A - The Institute of Education Sciences 

Sec. 111. Establishment. 

Sec. 112. Functions. 

Sec. 113. Delegation. 

Sec. 1 14. Office of the Director. 

Sec. 115. Priorities. 

Sec. 116. National Board for Education Sciences. 

Sec. 117. Commissioners of the National Education Centers. 

Sec. 118. Agreements. 

Sec. 119. Biennial report. 

Sec. 120. Competitive awards. 

Part B - National Center for Education Research 

Sec. 131. Establishment. 

Sec. 132. Commissioner for Education Research. 

Sec. 133. Duties. 

Sec. 134. Standards for conduct and evaluation of research. 

Part C - National Center for Education Statistics 

Sec. 151. Establishment. 

Sec. 152. Commissioner for Education Statistics. 

Sec. 153. Duties. 

Sec. 154. Performance of duties. 

Sec. 155. Reports. 

Sec. 156. Dissemination. 

Sec. 157. Cooperative education statistics systems. 

Sec. 158. State defined. 
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Part D - National Center for Education Evaluation and Regional Assistance 

Sec. 171. Establishment. 

Sec. 172. Commissioner for Education Evaluation and Regional Assistance. 
Sec. 173. Evaluations. 

Sec. 174. Regional educational laboratories for research, development, 
dissemination, and technical assistance. 

Part E - National Center for Special Education Research 

Sec. 175. Establishment. 

Sec. 176. Commissioner for Special Education Research. 

Sec. 177. Duties. 

Part F - General Provisions 

Sec. 181. Interagency data sources and formats. 

Sec. 182. Prohibitions. 

Sec. 183. Confidentiality. 

Sec. 184. Availability of data. 

Sec. 185. Performance management. 

Sec. 186. Authority to publish. 

Sec. 187. Vacancies. 

Sec. 188. Scientific or technical employees. 

Sec. 189. Fellowships. 

Sec. 190. Voluntary service. 

Sec. 191. Rulemaking. 

Sec. 192. Copyright. 

Sec. 193. Removal. 

Sec. 194. Authorization of appropriations. 

TITLE II - EDUCATIONAL TECHNICAL ASSISTANCE 

Sec. 201. Short title. 

Sec. 202. Definitions. 

Sec. 203. Comprehensive centers. 

Sec. 204. Evaluations. 

Sec. 205. Existing technical assistance providers. 

Sec. 206. Regional advisory committees. 

Sec. 207. Priorities. 

Sec. 208. Grant program for statewide, longitudinal data systems. 

Sec. 209. Authorization of appropriations. 

TITLE III - NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS 

Sec. 301. Short title. 

Sec. 302. National Assessment Governing Board. 

Sec. 303. National Assessment of Educational Progress. 

Sec. 304. Definitions. 

Sec. 305. Authorization of appropriations. 
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TITLE I - EDUCATION SCIENCES REFORM 

SEC. 101. SHORT TITLE. (20 USC 9501 note) 

This title may be cited as the "Education Sciences Reform Act of 2002". 

SEC. 102. DEFINITIONS. (20 USC 9501) 

In this title: 

(1) IN GENERAL. The terms "elementary school", "secondary school", "local 
educational agency", and "State educational agency" have the meanings given those 
terms in section 9101 of the Elementary and Secondary Education Act of 1965 

(20 U.S.C. 7801) and the terms "freely associated states" and "outlying area" have the 
meanings given those terms in section 1 121(c) of such Act (20 U.S.C. 6331(c)). 

(2) APPLIED RESEARCH. The term "applied research" means research — 

(A) to gain knowledge or understanding necessary for determining the means by 

which a recognized and specific need may be met; and 

(B) that is specifically directed to the advancement of practice in the field of 

education. 

(3) BASIC RESEARCH. The term "basic research" means research — 

(A) to gain fundamental knowledge or understanding of phenomena and 

observable facts, without specific application toward processes or products; and 

(B) for the advancement of knowledge in the field of education. 

(4) BOARD. The term "Board" means the National Board for Education Sciences 
established under section 116. 

(5) BUREAU. The term "Bureau" means the Bureau of Indian Affairs. 

(6) COMPREHENSIVE CENTER. The term "comprehensive center" means an entity 
established under section 203 of the Educational Technical Assistance Act of 2002. 

(7) DEPARTMENT. The term "Department" means the Department of Education. 

(8) DEVELOPMENT. The term "development" means the systematic use of 
knowledge or understanding gained from the findings of scientifically valid research and 
the shaping of that knowledge or understanding into products or processes that can be 
applied and evaluated and may prove useful in areas such as the preparation of 
materials and new methods of instruction and practices in teaching, that lead to the 
improvement of the academic skills of students, and that are replicable in different 
educational settings. 

(9) DIRECTOR. The term "Director" means the Director of the Institute of Education 
Sciences. 

(10) DISSEMINATION. The term "dissemination" means the communication and 
transfer of the results of scientifically valid research, statistics, and evaluations, in forms 
that are understandable, easily accessible, and usable, or adaptable for use in, the 
improvement of educational practice by teachers, administrators, librarians, other 
practitioners, researchers, parents, policymakers, and the public, through technical 
assistance, publications, electronic transfer, and other means. 
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(1 1 ) EARLY CHILDHOOD EDUCATOR. The term "early childhood educator" means 
a person providing, or employed by a provider of, nonresidential child care services 
(including center-based, family-based, and in-home child care services) that is legally 
operating under State law, and that complies with applicable State and local 
requirements for the provision of child care services to children at any age from birth 
through the age at which a child may start kindergarten in that State. 

(12) FIELD-INITIATED RESEARCH. The term "field-initiated research" means basic 
research or applied research in which specific questions and methods of study are 
generated by investigators (including teachers and other practitioners) and that conforms 
to standards of scientifically valid research. 

(13) HISTORICALLY BLACK COLLEGE OR UNIVERSITY. The term "historically 
Black college or university" means a part B institution as defined in section 322 of the 
Higher Education Act of 1965 (20 U.S.C. 1061). 

(14) INSTITUTE. The term "Institute" means the Institute of Education Sciences 
established under section 111. 

(15) INSTITUTION OF HIGHER EDUCATION. The term "institution of higher 
education" has the meaning given that term in section 101(a) of the Higher Education 
Act of 1965 (20 U.S.C. 1001(a)). 

(16) NATIONAL RESEARCH AND DEVELOPMENT CENTER. The term "national 
research and development center" means a research and development center 
supported under section 133(c). 

(17) PROVIDER OF EARLY CHILDHOOD SERVICES. The term "provider of early 
childhood services" means a public or private entity that serves young children, 
including — 

(A) child care providers; 

(B) Head Start agencies operating Head Start programs, and entities carrying out 

Early Head Start programs, under the Head Start Act (42 U.S.C. 9831 et seq.); 

(C) preschools; 

(D) kindergartens; and 

(E) libraries. 

(18) PRINCIPLES OF SCIENTIFIC RESEARCH - ‘ 

The term 'principles of scientific research' means research that - 



(A) applies rigorous, systematic, and objective methodology to obtain reliable 
and valid knowledge relevant to education activities and programs; 

(B) presents findings and makes claims that are appropriate to and supported 
by the methods that have been employed; and 



Congress has moved towards defining “principles of scientific research” rather than defining 
“scientifically-based research.” The differences, with one exception, aren’t substantive. This is 
Congress’s language as used in the Head Start reauthorization. 



Page 5 of 68 



(C) includes, appropriate to the research being conducted - 

(I) use of systematic, empirical methods that draw on observation or 
experiment; 

(II) use of data analyses that are adequate to support the general 
findings; 

(III) reliance on measurements or observational methods that provide 
reliable and generalizable findings; 

(IV) strong c l a i ms of causa l r el at i onsh i ps on l y i n r e s ea rch d e s i gns that 
eli m i n a t e p la us i b le comp e t i ng e xp la n a t i ons for obs e rv e d r e su l ts, 
wh i ch i nc l ud e but a r e not li m i t e d to r a ndom a ss i gnm e nt 
e xp e r i m e nts ;strong claims about an activity’s impact on educational 
outcomes only in well-designed and implemented random 
assignment experiments, when feasible, and other methods (such 
as well-matched comparison group studies) that allow for the 
strongest possible causal inferences when random assignment is 
not feasible * 

(V) presentation of studies and methods in sufficient detail and clarity 
to allow for replication or, at a minimum, to offer the opportunity to 
build systematically on the findings of the research; 

(VI) acceptance by a peer-reviewed journal or critique by a panel of 
independent experts through a comparably rigorous, objective, and 
scientific review; 

(VII) use of research designs and methods appropriate to the research 
questions posed; and 

(VIII) consistency of findings across multiple studies or sites to support 
the generality of results and conclusions. 



(19) SCIENTIFICALLY-VALID RESEARCH - The term 'scientifically valid 
research' includes applied research, basic research, and field-initiated research 
in which the rationale, design, and interpretation are soundly developed in 
accordance with accepted principles of scientific research 

SC I ENT I F I CALLY BASED RESEARCH STANDA RDS. 



This substitutes language adopted by the Board for Congress’s Head Start language. The Board 
language prioritizes randomized trials for causal claims whereas Congress’s language does not. 
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(A) Th e t e rm "sc ie nt i f i ca ll y bas e d r e s e arch standards" m e ans r e s e arch standards 
ttet — 



( i ) app l y r i gorous, sy s t e mat i c, and obj e ct i v e m e thodo l ogy to obta i n r eli ab le and 
va li d know le dg e r ele v a nt to e ducat i on act i v i t ie s and programs; and 

( ii ) pr e s e nt f i nd i ngs and mak e c l a i ms that ar e appropr i at e to and support e d by 
th e m e thods that hav e b ee n e mp l oy e d. 

(B) Th e t e rm i nc l ud e s, appropr i at e to th e r e s e arch b ei ng conduct e d — 

( i ) e mp l oy i ng syst e mat i c, e mp i r i ca l m e thods that draw on obs e rvat i on or 
e xp e r i m e nt; 

( ii ) i nvo l v i ng data ana l ys e s that ar e ad e quat e to support th e g e n e ra l f i nd i ngs; 

( ill ) r el y i ng on m e asur e m e nts or obs e rvat i ona l m e thods that prov i d e r eli ab le 
data; 

( i v) mak i ng c l a i ms of causa l r el at i onsh i ps on l y i n random ass i gnm e nt 
e xp e r i m e nts or oth e r d e s i gns (to th e e xt e nt such d e s i gns substant i a ll y eli m i nat e 
p l aus i b le comp e t i ng e xp l anat i ons for th e obta i n e d r e su l ts); 

(v) e nsur i ng that stud ie s and m e thods ar e pr e s e nt e d i n suff i c ie nt d e ta il and 
c l ar i ty to a ll ow for r e p li c a t i on or, a t a m i n i mum, to off e r th e opportun i ty to bu il d 
syst e mat i ca ll y on th e f i nd i ngs of th e r e s e arch; 

(v i ) obta i n i ng acc e ptanc e by a p ee r - r e v ie w e d journa l or approva l by a pan el of 
i nd e p e nd e nt e xp e rts through a comparab l y r i gorous, obj e ct i v e , and sc ie nt i f i c 
r e v ie w; and 

(v ii ) us i ng r e s e arch d e s i gns and m e thods appropr i at e to th e r e s e arch qu e st i on 
pos e d. 

(204^) SCIENTIFICALLY VALID EDUCATION EVALUATION. The term "scientifically 
valid education evaluation" means an evaluation that — 

(A) adheres to the highest possible standards of quality with respect to research 
design and statistical analysis; 

(B) provides an adequate description of the programs evaluated and, to the extent 
possible, examines the relationship between program implementation and program 
impacts; 

(C) provides an analysis of the results achieved by the program with respect to its 
projected effects; 

(D) employs experimental designs using random assignment, when feasible, and 
other research methodologies that allow for the strongest possible causal inferences 
when random assignment is not feasible; and 

(E) may study program implementation through a combination of scientifically valid 
and reliable methods. 

(20) SC I ENT I F I CALLY VAL I D RESEARCH. Th e t e rm "sc ie nt i f i ca ll y va li d r e s e arch" 
i nc l ud e s app lie d r e s e arch, bas i c r e s e arch, and f iel d -i n i t i at e d r e s e arch i n wh i ch th e 
rat i ona le , d e s i gn, and i nt e rpr e tat i on ar e sound l y d e v el op e d i n accordanc e w i th 
sc ie nt i f i ca ll y bas e d r e s e arch standards. 

(21 ) SECRETARY. The term "Secretary" means the Secretary of Education. 
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(22) STATE. The term "State" includes (except as provided in section 158) each of 
the 50 States, the District of Columbia, the Commonwealth of Puerto Rico, the freely 
associated states, and the outlying areas. 

(23) TECHNICAL ASSISTANCE. The term "technical assistance" means — 

(A) assistance in identifying, selecting, or designing solutions based on research, 
including professional development and high-quality training to implement solutions 
leading to — 

(i) improved educational and other practices and classroom instruction based 
on scientifically valid research; and 

(ii) improved planning, design, and administration of programs; 

(B) assistance in interpreting, analyzing, and utilizing statistics and evaluations; 
and 

(C) other assistance necessary to encourage the improvement of teaching and 
learning through the applications of techniques supported by scientifically valid 
research. 



PART A - THE INSTITUTE OF EDUCATION SCIENCES 

SEC. 111. ESTABLISHMENT. (20USC9511) 

(a) ESTABLISHMENT. There shall be in the Department the Institute of Education 
Sciences, to be administered by a Director (as described in section 1 14) and, to the 
extent set forth in section 1 16, a board of directors. 

(b) MISSION. 

(1) IN GENERAL. The mission of the Institute is to provide national leadership in 
expanding reliable evidence on which to ground education practice and policy and to 
encourage its use prov i d e nat i ona l le ad e rsh i p i n e xpand i ng fundam e nta l know le dg e 
and und e rstand i ng of e ducat i on from e ar l y ch il dhood through posts e condary study, i n 
ord e r to prov i d e by parents, educators, students, researchers, policymakers, and the 
general public. The Institute shall provide information about w i th r eli ab le i nformat i on 
about — 



(A) the condition and progress of education in the United States, including early 
childhood education and special education; 

(B) educational practices that support learning and improve academic 
achievement and access to educational opportunities for all students; and 

(C) the effectiveness of Federal and other education programs. 

(2) CARRYING OUT MISSION. In carrying out the mission described in 
paragraph (1), the Institute shall compile statistics, develop products, and conduct 
research, evaluations, and wide dissemination activities in areas of demonstrated 



The new definition, which focuses on providing evidence and encouraging its use, is much 
closer to what lES does and is more objective than the existing definition’s focus on fundamental 
knowledge and understanding. 
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national need (including in technology areas) that are supported by Federal funds 

appropriated to the Institute and ensure that such activities — 

(A) conform to high standards of quality, integrity, and accuracy; and 

(B) are objective, secular, neutral, and nonideological and are free of partisan 
political influence and racial, cultural, gender, or regional bias. 

(c) ORGANIZATION. The Institute shall consist of the following: 

(1) The Office of the Director (as described in section 114). 

(2) The National Board for Education Sciences (as described in section 116). 

(3) The National Education Centers, which include — 

(A) the National Center for Education Research (as described in part B); 

(B) the National Center for Education Statistics (as described in part C); ae4 

(C) the National Center for Education Evaluation and Regional Assistance (as 
described in part D)t; and 

(D) the National Center for Special Education Research (as described in 
part E). 

SEC. 112. FUNCTIONS. (20 USC 9512) 

From funds appropriated under section 194, the Institute, directly or through grants, 
contracts, or cooperative agreements, shall — 

(1) conduct and support scientifically valid research activities, including basic 
research and applied research, statistics activities, scientifically valid education 
evaluation, development, and wide dissemination; 

(2) widely disseminate the findings and results of scientifically valid research in 
education; 

(3) promote the use, development, and application of knowledge gained from 
scientifically valid research activities; 

(4) strengthen the national capacity to conduct, develop, and widely disseminate 
scientifically valid research in education; 

(5) promote the coordination, development, and dissemination of scientifically valid 
research in education within the Department and the Federal Government; and 

(6) promote the use and application of research and development to improve practice 
in the classroom. 

SEC. 113. DELEGATION. (20USC9513) 

(a) DELEGATION OF AUTHORITY. Notwithstanding section 412 of the Department 
of Education Organization Act (20 U.S.C. 3472), the Secretary shall delegate to the 
Director all functions for carrying out this title (oth e r than adm i n i strat i v e and support 
funct i ons) , except that — 

(1) the Institute shall utilize the Department’s administrative support services for 
travel, contracting, purchasing, human resources, information technology. 
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and facilities under terms established by the Secretary that do not infringe 
on the Director’s responsibilities as detailed in this Act;* 

(2) nothing in this title or in the National Assessment of Educational Progress 
Authorization Act (except section 302(e)(1)(J) of such Act) shall be 
construed to alter or diminish the role, responsibilities, or authority of the 
National Assessment Governing Board with respect to the National 
Assessment of Educational Progress (including with respect to the 
methodologies of the National Assessment of Educational Progress 
described in section 302(e)(1)(E)) from those authorized by the National 
Education Statistics Act of 1994 (20 U.S.C. 9001 et seq.) on the day before 
the date of enactment of this Act; 



(2) m e mb e rs of th e Nat i ona l Ass e ssm e nt Gov e rn i ng Board sha ll cont i nu e to b e 
appo i nt e d by th e S e cr e tary; 

(3) s e ct i on 302(f)(1 ) of th e Nat i ona l Ass e ssm e nt of Educat i ona l Progr e ss 
Author i zat i on Act sha ll app l y to th e Nat i ona l Ass e ssm e nt Gov e rn i ng Board i n th e 
e x e rc i s e of i ts r e spons i b ili t ie s und e r th i s Act; 

(4) s e ct i ons 115 and 116 sha ll not app l y to th e Nat i ona l Ass e ssm e nt of 
Educat i ona l Progr e ss; and 

(5) s e ct i ons 115 and 116 sha ll not app l y to th e Nat i ona l Ass e ssm e nt Gov e rn i ng 
Board 

(b) OTHER ACTIVITIES. The Director may accept from the * Secretary-may 
assignments for the Institute to r e spons i b ili ty for administermg other activities, if those 
activities are consistent with — 

(1) the Institute's priorities, as approved by the National Board for Education 
Sciences under section 116, and the Institute's mission, as described in section 
111(b); or 

(2) the Institute's mission, but only if those activities do not divert the Institute from 
its priorities. 



Determining exactly what authority is delegated to the Director and what remains with the 
Secretary has been difficult due to the ambiguity of the current phrase, “other than administrative 
and support functions”. For example, do “administrative functions” reserve to the Secretary the 
authority to determine which particular staff are hired by the Director or is this the Director’s 
responsibility.? Do “administrative functions” include managing contact with the press or does the 
Director have the authority to interact with the press without approval of the Secretary? Do 
“administrative functions” include requiring that annual employee performance plans within lES 
include the Secretary’s priorities? The new definition is more precise in providing a list of specific 
functions that are reserved to the Department. 

NAGB had sections 2, 3, 4, & 5 inserted into ESRA because of their concern that someone 
would interpret ESRA as diminishing the role of NAGB. This principle is clearly articulated in the 
paragraph just prior to these sections, making these sections unnecessary. 

** This change makes it lES’s option to accept additional assignments from the Secretary. The 
current language makes it possible for the Secretary to assign activities to lES that may not be 
appropriate or consistent with lES’s priorities or mission since it would be the Secretary who 
would make the determination of consistency. 
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SEC. 114. OFFICE OF THE DIRECTOR. (20 USC 9514) 

(a) APPOINTMENT. Exc e pt as prov i d e d i n subs e ct i on (b)(2), t The President, by and 
with the advice and consent of the Senate, shall appoint the Director of the Institute. 

(b) TERM. 

(1) IN GENERAL. The Director shall serve for a term of 6 years, beginning on 
the date of appointment of the Director, except that the Director may serve 
after the expiration of the Director's term, until a successor has been 
appointed, for a period not to exceed 1 additional year^- 

(2) RENOMINATION. A Director may be renominated for a new term, * 

(2) F I RST D I RECTOR. Th e Pr e s i d e nt, w i thout th e adv i c e and cons e nt of th e 
S e nat e , may appo i nt th e Ass i stant S e cr e tary for th e Off i c e of Educat i ona l R e s e arch 
a nd I mprov e m e nt ( a s such off i c e e x i st e d on th e day b e for e th e dat e of e nactm e nt of 
th i s Act) to s e rv e a s th e f i rst D i r e ctor of th e I nst i tut e . 

(3) BOARD RECOMMENDATIONS. The Board may make recommendations to 
the President with respect to the appointment of a Director under subsection (a)otber 
than a D i r e ctor appo i nt e d und e r paragraph (2). 

(c) PAY. The Director shall receive the rate of basic pay for level II of the Executive 
Schedule and shall be eligible for critical pay under the Federal Workforce Flexibility Act 
of 2004.*" 

(d) QUALIFICATIONS. The Director shall be selected from individuals who are highly 
qualified authorities in the fields of scientifically valid research, statistics, or evaluation in 
education, as well as management within such areas, and have a demonstrated capacity 
for sustained productivity and leadership in these areas. 

(e) ADMINISTRATION. The Director shall— 

(1 ) administer, oversee, and coordinate the activities carried out under the 
Institute, including the activities of the National Education Centers; ae4 

(2) coordinate and approve budgets and operating plans for each of the 
National Education Centers for submission to the Secretary; and 

(3) report directly to the Secretary.**" 

(f) DUTIES. The duties of the Director shall include the following: 

(1) To propose to the Board priorities for the Institute, in accordance with 
section 11 5(a). 

(2) To submit to the Board every two years a biennial plan of activities 
consistent with the priorities under Sec. 1 1 5. 



Enhances the possibility of continuity in leadership when a director’s term expires but no-one 
has been nominated to replace the director. 

Makes explicit what could be interpreted as permissible in current law. 

*** Many people who might be qualified to be Director are unwilling to do so over a 6 year term at 
the rate of pay of level II. This addition provides pay flexibility in recruiting a director, and would 
be subject to the recommendation of the Board. 

**" A direct reporting line to the Secretary is important to maintaining the status and independent 
functioning of lES within the Department 



Page 11 of 68 



